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Executive  Summary 


The  undeniable  importance  of  large  high-definition  displays  accompanied  by 
advanced  techniques  for  effective  human-computer  interaction  (HCI)  that  are 
suitable  for  Command  and  Control  (C2)  environments  are  paramount  to  the 
success  of  satisfying  the  Air  Force’s  information  management  requirements. 
Airmen  are  constantly  faced  with  making  key  decisions  utilizing  massive  amounts 
of  information,  which  is  not  only  dynamic,  but  originating  from  multiple  sources 
including  various  military  applications,  sensors,  databases,  live  satellite  input, 
and  video  feeds. 

The  Advanced  Displays  and  Intelligent  Interfaces  (ADII)  program  objectives  and 
accomplishments  include  the  evaluation,  exploitation,  development,  and 
advancement  of  display  and  vital  man-machine  interaction  technologies.  As  a 
result  the  Interactive  DataWall  has  emerged  as  an  operational  product  that  is  and 
continues  to  be  a  proving  ground  for  new  concept  development  and 
experimentation  in  these  technology  areas. 

Techniques  have  been  developed  that  leverage  commercial-off-the-shelf  (COTS) 
display  technology  to  provide  high-resolution  large-screen  displays  by  effectively 
tiling  multiple  video  projectors  in  a  near  seamless  3x1  matrix.  Custom  support 
structures  have  been  designed  and  fabricated  to  provide  a  mechanism  to 
precisely  align  the  independent  display  components  to  create  a  composite  high- 
resolution  display. 

Information  can  be  displayed  on  the  DataWall  in  number  of  ways  that  range  from 
running  local  on  the  its  central  computer  system,  over  the  network  from  remote 
systems,  or  utilizing  COTS  video  processing  hardware  with  custom  developed 
software.  All  provide  different  levels  of  performance  in  terms  of  display  refresh 
and  interactive  control. 

Wireless  interaction  is  accomplished  through  speaker-independent  speech 
recognition  and  an  in-house  developed  laser  pointer  input  device.  Live  video  of 
the  display  surface  is  processed  to  determine  the  presence  and  specific  position 
of  a  laser  dot  on  the  screen  to  allow  operators  to  interact  with  the  information 
displayed.  The  laser  pointer  interface  provides  a  mouse-like  mode  of  input  for  a 
very  natural  and  unencumbered  means  of  data  manipulation. 

The  usability  and  portability  for  field  deployments  of  the  systems  under 
development  has  been  a  key  factor  most  recently.  The  decision  maker’s  role 
must  not  to  be  impaired  by  the  mechanics  of  utilizing  the  tools  available,  but  one 
that  will  facilitate  better  information  management  and  an  improved  situational 
awareness  to  fulfill  his/her  mission.  A  significant  amount  of  research  and 
development  has  been  accomplished  to  package  the  Interactive  DataWall  into 
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one  that  is  field  deployable.  It  has  evolved  into  a  system  with  more  simplified  set¬ 
up  procedures  and  a  significantly  reduced  deployable  footprint. 

Several  systems  have  been  transitioned  to  sites  outside  the  laboratory  including 
to  warfighters  in  theater  operations.  The  feedback  from  real  users  has  been  an 
invaluable  resource  to  steer  the  development  of  our  interactive  display  systems. 

Future  research  and  development  for  the  Interactive  DataWall  includes  the 
integration  of  touch  and  gesture  as  additional  modes  of  input.  Also  planned  are 
improvements  to  the  display  structures  to  further  improve  usability  in  the  field 
including  automatic  projector  and  camera  alignment  mechanisms. 
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1  Introduction 


Solving  the  challenges  facing  the  21st  century  commander  in  dominating  the 
information  management  battlefield  heavily  correlates  with  the  assimilation, 
analysis  and  strong  interaction  with  not  only  an  extensive  amount  of  data  and 
information,  but  that  which  is  also  detailed,  dynamic,  variable  in  format,  and 
originating  from  multiple  channels  and  resources.  The  immense  amount  of 
information  encompasses  sources  and  elements  such  as  terrain  and  geographic 
area  representations,  land  route  maps,  database  information,  intelligence  and 
surveillance  collections,  graph  analysis,  target  modeling  and  simulation  results. 
In  addition,  increased  tempo,  higher  precision,  and  more  complex  data  sources 
have  increased  the  demand  for  tailored  information  and  have  increased 
emphasis  on  revolutionary  information  technologies.  Paramount  to  the 
aggregation  of  information  are  the  display  system’s  accessibility  and  real  time 
performance,  especially  in  handling  streaming  video  and  audio  inputs,  and  while 
running  highly  intensive  graphic  2D  and  3D  renditions.  The  ADII  team  recognizes 
the  requirements  and  the  importance  of  addressing  the  interdependent  and 
intense  issues  challenging  today’s  decision  makers. 

The  development  of  a  massive,  high  resolution  tiled  display  has  the  ability  to 
augment  situational  awareness  for  an  entire  C2  audience.  The  uniqueness  of  the 
Interactive  DataWall  (IDW)  is  the  push  to  actively  foster  collaboration  and 
participation  among  a  collection  of  decision  makers.  Multiple  users  can 
simultaneously  interact  with  the  display,  including  those  in  co-located 
environments.  Additionally  enhancive  are  the  multiple  methods  available  to 
operators  and  decision  makers  alike  for  unencumbered  control  of  the  display 
workspace,  via  camera-tracked  laser  pointers,  and  wireless  speech  interaction. 

Management  of  display  development  by  the  ADII  team  extends  beyond  the 
demands  associated  with  massive,  high  resolution  display  systems  and 
enhancing  software  functionality.  Equal  consideration  is  given  to  the  conditions 
and  logistics  associated  with  utilizing  this  display  technology.  Military  exercises 
and  operations  require  integrated  systems  and  software  that  need  to  be  robust 
and  functionally  easy  to  setup.  Moreover,  such  systems  must  be  capable  of 
withstanding  extreme  external  elements  and  environmental  restrictions  while 
faced  with  the  sometimes  limited  resources  available  to  the  warfighter.  In 
essence,  an  ideal  display  system  would  be  as  cognitively  and  visually  effective 
as  it  is  physically  capable.  In  addition,  it  would  provide  a  large  interactive 
workspace,  be  flexible  during  transportation  and  deployment,  and  have  minimal 
footprint. 
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2  Interactive  Data  Wall 


2.1  Rationale 

A  tiled  display  is  a  matrix  of  multiple  display  devices  such  as  Liquid  Crystal 
Displays  (LCDs),  Cathode  Ray  Tube  (CRT)  monitors,  plasma  screens,  or  video 
projectors  positioned  next  to  each  other,  to  create  a  single  continuous  image. 
The  approach  of  using  a  large  tiled  display  such  as  the  Interactive  DataWall 
(IDW)  has  evident  advantages  in  enabling  mission  planning  in  today’s  battlefield. 
Primarily,  a  large  tiled  display  offers  more  available  workspace  for  viewing  data 
and  information.  In  critical  C2  situations,  enabling  comparisons  and  relationships 
across  different  applications  is  critical  to  decision-making.  Moreover,  a  larger  tiled 
display  is  accompanied  by  an  increase  in  image  resolution,  which  leads  to  better 
image  scalability.  Higher  levels  of  detail  are  available  to  the  commander 
improving  the  viewing  clarity  and  ability  to  manage  applications.  This  is  especially 
true  for  those  requiring  high  performance  such  as  graphical  maps  and  high 
resolution  satellite  imagery.  A  large  tiled  display  comprising  of  high-resolution 
devices  provide  the  avenue  for  exposing  deeper  levels  of  imagery  pertinent  in 
analyzing  and  assessing  essential  details  in  visualized  data.  Furthermore,  larger 
displays  enable  more  effective  collaboration  within  a  localized  working 
environment.  Tasks  where  missions  are  planned  and  discussed  necessitate 
information  that  is  effortlessly  shared  with  an  entire  audience. 


2.2  Challenges 

Attaining  a  considerably  expanded  display  area  is  bounded  by  several 
fundamental  technology  challenges.  The  goal  is  to  compose  a  perceptually  whole 
viewable  area  utilizing  a  collection  of  individual  display  devices.  Instrumental  to 
constructing  this  single  near-seamless  display  workspace  includes  optimally 
weighing  choices  for  the  components  to  minimize  the  disparity  among  tiles,  both 
physical  and  image-related. 

Approaches  for  both  precise  manual  and  automatic  alignment  for  projection- 
based  display  systems  are  available  to  greatly  reduce  the  visibility  of  seams  and 
gaps  between  image  tiles,  but  each  with  its  own  limitations  and  trade-offs.  Color 
and  brightness  variations  across  projected  images  within  tiled  displays  can  also 
cause  an  inhomogeneous  appearance  to  the  imagery.  What  has  to  be 
considered  in  implementing  solutions  to  these  problems  is  what  is  acceptable  in 
terms  of  image  distortion,  ease  of  adjustments,  and  the  constraints  imposed  by 
operational  environment  conditions. 
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The  creation  of  a  large  display  space  also  introduces  new  challenges  for 
effectively  managing  and  interacting  with  the  information  displayed.  Conventional 
input  devices  designed  for  desktop  use  are  limiting  and  often  cumbersome  to  use 
in  large-screen  environments. 

The  Interactive  DataWall  system  architecture  was  designed  cognizant  of 
technology  limitations  and  advantages  to  integrate  the  most  optimal  choices  for 
display  devices,  screen  materials,  computer  systems,  video  processing 
hardware,  projector  support  and  alignment  systems,  display  software,  and 
interactive  techniques.  All  are  essential  to  create  an  optimal  display  system 
devised  for  today’s  decision  maker. 


2.3  System  Architecture 

The  Interactive  DataWall  (IDW)  consists  of  three  LCD-based  video  projectors, 
each  with  a  resolution  of  1280  x  1024  pixels.  They’re  mounted  in  a  horizontal 
configuration  projecting  onto  a  permanently  installed  12'  x  3%'  display  screen 
resulting  in  a  near-seamless  single  image.  The  current  configuration  has  a  total 
display  resolution  of  approximately  3.9  million  pixels  (3840  x  1024)  over  the 
display  area  (Figure  1 ). 

The  IDW  leverages  vital  COTS  software  and  hardware  components  to  drive  its 
multi-screen  extended  desktop.  All  instances  of  the  IDW  through  its  evolution 
have  been  driven  by  a  single  central  computer  system  configured  with  either 
multiple  graphics  cards  or  multi-output  graphics  cards.  Each  output  provides  a 
high-resolution  source  for  each  tile  of  the  composite  DataWall  display.  The 
creation  of  this  meta-desktop  allows  windows  to  be  repositioned  and  resized 
anywhere  on  the  display.  Projecting  windows  originating  from  multiple  sources 
across  these  boundaries  is  of  particular  importance  through  the  availability  of  the 
Microsoft®  meta-desktop  capability.  Windows  can  also  be  maximized  to  fill  the 
entire  display.  Many  application  windows  can  be  run  simultaneously  on  the 
DataWall  computer  with  windows  of  current  interest  displayed  in  the  foreground, 
positioned  to  the  background  or  minimized  for  quick  accessibility  when  needed. 

Early  IDW  versions  used  high-end  graphics  workstations  with  special  COTS 
software  to  create  a  very  high-resolution  workspace.  During  the  latter  iterations, 
Personal  Computers  (PCs)  have  been  the  primary  DataWall  platform  due  to  a 
shift  in  cost  and  software  support  considerations.  Several  graphics  card 
manufacturers  capitalized  on  the  meta-desktop  approach  and  Microsoft® 
Windows  widespread  use  by  developing  special  drivers  for  earlier  versions  of 
Windows.  In  turn,  Microsoft®  recognized  the  benefit  of  this  approach  and 
provided  built-in  meta-desktop  support  since  the  release  of  Windows  2000.  The 
utility  of  a  meta-desktop  to  manage  a  user’s  information  space  is  being 
recognized  as  users  have  begun  to  adopt  the  practice  on  their  desktop  PCs 
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using  multiple  monitors.  The  DataWall  improves  on  the  notion  by  increasing  the 
scale  for  larger  group  viewing  and  providing  a  seamless,  less  fragmented  display 
space. 


Figure  1  Interactive  DataWall  System  Architecture 

A  vital  part  of  the  IDW  is  the  ability  not  only  to  display  PC  driven  applications,  but 
to  accept  multiple  inputs  from  various  sources.  The  IDW  architecture  supports 
the  ability  to  accept  multiple  live,  dynamic  video  streams  and  displaying  them  in 
any  of  n  x  m  configured  tiles.  Achievement  of  this  feature  is  through  the 
incorporation  of  three  external  video  processors  connected  to  the  central 
DataWall  PC  system. 
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The  extended  display  space  enables  the  capability  of  a  collaborative  display 
environment.  The  latter  iterations  of  DataWalls  render  the  function  for  a  number 
of  decision  makers  to  interact  within  the  environment  simultaneously  via  multiple 
cursors  generated  by  a  collection  of  networked  application  computers.  Co- 
located  application  computers  can  also  be  controlled  within  the  display  space  as 
well. 


3  Capabilities 


3.1  Displaying  Information 

3.1.1  Running  Local  on  DataWall  Computer 

One  method  for  displaying  information  on  the  DataWall  is  to  have  the  application 
software  installed  and  run  on  the  central  DataWall  computer  system.  Running 
applications  locally  benefits  from  the  ability  to  manipulate  the  applications  with  a 
variety  of  wireless  input  devices  available.  This  is  an  important  distinction 
between  the  centralized  DataWall  computer  approach  and  the  more  commonly 
used  video  switching  hardware  approach.  Even  if  only  conventional  keyboards 
and  mice  are  utilized,  the  operators  have  control  over  the  applications  being 
displayed  rather  than  the  display  serving  only  as  a  summary  device. 

This  approach  also  benefits  applications  requiring  optimal  processing  power  and 
frequent  refreshes  to  run  efficiently  such  as  graphics  intensive  visualizations.  All 
IDW  central  computer  systems  have  been  customized  with  the  latest  state-of-the- 
art  components  during  development.  Though  highly  favorable  for  typical  install- 
and  run-programs,  a  fair  proportion  of  Command  and  Control  scenarios  may 
determine  the  described  resident  approach  as  not  necessarily  the  most  practical. 
In  many  cases  it  is  more  efficient  to  run  and  display  applications  from  remote 
computers  because  of  software  installation  restrictions,  and  practical  limits  of  any 
computer’s  processing  power  to  run  multiple  applications  simultaneously. 


3.1.2  Network-Based  Control  Software 

Developed  with  the  University  of  Alabama,  through  a  program  entitled, 
“Enhanced  Interactive  DataWall  for  Collaboration  and  Data  Fusion”,  collaborative 
software  enables  external  systems  reachable  by  existing  network  connectivity  to 
be  displayed.  The  software  utilizes  a  VNC-based  (Virtual  Network  Computing) 
framework  and  modified  calls  to  allow  applications  running  though  this  procedure 
to  be  manipulated  simultaneously  on  the  DataWall.  Multiple  cursors  operating  at 
the  same  time  are  available  for  users  to  work  simultaneously  across  different 
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applications.  Local  application  computers  can  push  information  to  the  DataWall 
directly  utilizing  a  local  network  hub  offering  fast  refreshes  for  the  sources  being 
displayed  on  the  DataWall  workspace.  If  global  high-bandwidth  network 
connectivity  is  available,  or  applications  are  required  to  be  displayed  that  are 
more  remotely  located,  they  can  also  be  displayed  utilizing  the  same  software. 
For  this  approach,  network  bandwidth  and  availability  is  the  primary  limiting  factor 
for  sending  and  receiving  data. 


3.1.3  Client-Server  Coupling 

Another  alternative  is  a  client-server  based  arrangement.  The  program  with  much 
lower  overhead  client  software  can  run  on  the  DataWall  computer  while  the 
server  runs  elsewhere  on  the  network.  This  is  a  very  efficient  means  of  displaying 
on  the  DataWall  screen,  but  it  requires  predisposed  client-server  software. 


3.1.4  Fenestra 

A  proportion  of  military  collaboration  involves  sources  that  originate  from  external 
devices  such  as  laptops  and  video  devices  that  provide  both  recorded  and  live 
video  feeds  that  need  to  be  displayed  in  real-time.  A  display  that  fails  to  refresh 
at  a  comparable  rate  of  the  original  source  is  uncomfortable  to  view  and  fails  to 
meet  the  real-time  requirement  for  mission  critical  data  display.  The  DataWall 
system  also  incorporates  a  system  called  Fenestra  (which  means  windows  in 
Latin)  that  leverages  COTS  video  overlay  hardware  with  custom  in-house 
developed  windowing  software  to  display  live  high-resolution  computer  video  and 
low-resolution  NTSC  (National  Television  System  Committee)  video  sources.  It 
provides  a  means  to  manage  external  video  sources  within  any  tiled  high- 
resolution  display  in  real  time,  unimpaired  by  refresh  latency  characteristic  of 
network-based  approaches  with  slow  connectivity.  In  addition  to  the  ability  to 
display  multiple  external  sources  in  real-time  it  also  has  the  advantages  of  not 
requiring  network  connectivity  nor  previous  installation  of  additional  software.  The 
video  windows  can  be  scaled  and  oriented  anywhere  on  the  display  screen, 
together  with  other  DataWall  controller  tools.  If  network  bandwidth  is  unavailable 
and  conditions  do  not  permit  installing  and  running  applications  on  the  DataWall 
PC,  it  is  an  optimal  alternative. 


3.1. 4.1  RGB  Spectrum  SuperView™  Hardware  Description 

A  critical  component  of  the  Fenestra  architecture  is  the  SuperView™  3000RT 
from  RGB  Spectrum.  It  is  a  COTS  video  processing  system  that  can  be 
configured  to  display  up  to  twelve  real-time  video  and  computer  signals  on  a 
single  high  resolution  screen.  Sources  appear  as  borderless  windows  on  the 
display  space.  It  accepts  NTSC/PAL  composite  video,  Y/C  (S-Video),  and 
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component  YUV  signals  from  cameras,  tape  recorders,  videodisc  or 
teleconferencing  systems,  and  high  line  rate  video  signals  from  computers, 
FLIRs,  medical  imagers,  and  radar  with  resolutions  up  to  1600  x  1200  pixels. 
Each  input  can  be  positioned,  scaled  to  any  size,  overlaid  with  computer  graphics 
or  overlapped  with  other  signals.  In  addition,  the  user  can  pan  and  zoom  within 
each  image.  Control  for  manipulation  of  these  windows  is  through  the 
manufacturer’s  software  package  via  RS-232  serial  commands,  or  through  an 
optional  front  control  panel. 


3.1. 4.1.1  Advantages 

Real-Time  Performance  -  Compared  to  other  methods  of  displaying  external 
sources,  the  SuperView™  hardware  can  display  high-quality,  real-time  video 
information  and  graphics  on  a  large  screen  display  without  significant  processing 
impact.  The  rationale  for  the  remarkable  performance  is  all  video  processing  is 
segregated,  and  the  SuperView™  serves  as  a  pass-through  device.  Although  the 
availability  and  popularity  of  TV  tuner  cards  are  more  widespread  and  could  be 
construed  as  an  alternative,  integrated  cards  suffer  from  the  limitations  of  utilizing 
shared  processing  resources  with  the  primary  system.  The  notion  of  isolating  all 
video  processing  into  a  separate  system  provides  optimal  tasking  of  delivering 
real-time  information,  videos,  and  images  at  a  comfortable  speed,  void  of 
noticeable  lag  during  operator  use.  It  also  provides  the  capability  to  display  high- 
resolution  computer  signal  that  a  traditional  TV  tuner  cannot  support. 

Dynamic  Background  -  Compared  to  other  video  processing  systems,  the 
SuperView™  is  capable  of  overlaying  the  primary  computer  system’s  background 
as  a  separate  input.  Additional  sources  are  secondary  overlays  to  the  desktop. 
The  setup  enables  the  use  of  the  primary  system’s  resources,  such  as  installed 
applications  and  saved  files  to  be  launched  together  with  live  video  windows  from 
external  sources.  Commonly  in  similar  systems,  the  background  is  a  static 
image,  and  only  external  video  windows  can  be  displayed,  thus  limiting  the 
primary  function  of  just  being  able  to  view  multiple  input  sources. 


3.1. 4.1. 2  Disadvantages 

Although  performance  for  rendering  intensive  graphics  and  videos  is  effectively 
satisfied  by  the  SuperView™,  there  are  limitations  to  its  design. 

Single  Display  Use  -  Although  multiple  sources  can  be  projected  for  viewing  and 
manipulation,  the  machine’s  design  primarily  caters  to  outputting  to  a  single 
display  device.  In  common  C2  environments,  large  tiled  displays  such  as  the 
DataWall  are  utilized  to  display  and  manage  the  information.  Restricting  the  live 
windows  to  one  section  of  the  screen  out  of  n  image  tiles  available  is 
cumbersome  and  limiting  to  the  operator,  or  the  group  of  decision-makers  who 
may  wish  to  place  windows,  and  manage  the  entire  viewable  space  during 
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presentations  and  discussions.  A  version  of  the  software  is  available  for  limited  2- 
tile  window  movement,  however  testing  of  the  software  during  demonstrations 
revealed  reliability  issues,  and  a  complicated  user  interface. 

Interface  Complexity  -  The  manufacturer’s  accompanying  software  for  display 
management  imposes  a  high  learning  curve  to  use  effectively.  Considerable 
background  knowledge  is  required  in  video  processing  procedures  such  as  host 
and  input  timing  parameters,  and  other  computer  and  video  domain-specific 
adjustments.  For  C2  environments,  the  task  of  the  decision-maker  or  group  of 
decision-makers  is  to  utilize  the  videos  and  input  resources  to  make  determinate 
and  knowledgeable  decisions,  and  not  to  be  impaired  with  the  process  of  using 
the  technology  (Figure  2). 


Figure  2  Manufacturer’s  Virtual  Screen  Panel 

Indirect  Window  Interaction  -  As  mentioned  earlier,  live  video  and  computer 
inputs  can  be  displayed  via  two  methods.  One  is  through  the  manufacturer’s 
supplied  software  interface,  and  the  other  is  through  the  front  panel,  an  additional 
cost  option  provided  in  some  SuperView™  models.  The  pitfall  behind  this 
implementation  stems  from  the  unnatural  way  of  managing  multiple  windows. 
Both  methods  require  the  operator  to  cope  and  interact  with  source  manipulation 
tools  indirectly,  and  to  map  results  against  the  display.  As  an  example,  to  resize 
or  move  a  particular  window,  sequenced  steps  from  a  multiple  layered  menu 
need  to  be  traversed  in  order  to  push  the  correct  parameter  button. 

If  the  method  is  through  the  Virtual  Screen  Panel,  live  window  interaction  is 
collateral.  Users  rely  on  feedback  to  verify  movement  within  the  white  area  by 
mapping  events  to  the  actual  display  work  space.  The  Virtual  Screen  panel  uses 
this  map  and  verifies  technique  for  moving  and  resizing  windows,  instead  of  a 
direct  approach  of  managing  windows  on  the  display  itself. 

Timing  Adjustments  -  The  SuperView™  framework  relies  on  the  concepts  of 
ordering  overlays.  Although  the  system  technically  detects  the  timing  parameters 
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of  each  overlay,  more  often  than  not,  further  adjustments  for  accurate  calibration 
are  necessary.  The  background  overlay,  usually  the  primary  computer’s  desktop, 
needs  to  be  adjusted  to  map  the  display  device’s  coordinate  system.  Tiled 
displays  pose  an  added  degree  of  complication  since  the  calibration  is  performed 
n  times,  where  n  is  the  number  of  display  devices.  Similarly,  input  sources 
displayed  as  windows  are  treated  as  a  second  layer  of  overlays,  and  require 
procedures  for  calibration  as  well.  The  method  for  interacting  with  the  display  for 
acquiring  correct  timings  demand  additional  operator  learning,  compounded  by 
the  interface  complexities  mentioned  previously. 


3.1. 4.2  Video  Overlay  Requirements  for  DataWall  Integration 

The  primary  objective  was  to  provide  operators  with  the  ability  to  effectively 
display  high  quality,  real-time  computer  and  low-resolution  video  from  external 
sources  onto  the  DataWall.  The  COTS  video  overlay  approach  discussed 
previously  needed  to  be  enhanced  for  tiled  display  systems  and  a  needed  a 
simplified  user  interface. 

The  original  DataWall  arrangement  simply  piped  each  of  three  SuperView™ 
outputs  to  one  of  three  projectors  comprising  the  DataWall.  The  SuperView’s™ 
design  limited  viewing  and  managing  windows  within  the  perimeter  of  an  image 
tile,  corresponding  to  a  single  display  device  (Figure  3).  A  significant  challenge 
was  extending  the  wall  control  functionalities  across  a  tiled  display,  such  as  the 
DataWall. 


Figure  3  Original  Video  Window  Configuration 

User  interaction  for  window  management  also  needed  to  be  direct  and 
transparent.  Complications  regarding  the  hardware  setup  and  the  complexities  of 
the  SuperView™  operations  needed  to  be  hidden  from  the  user.  The  most 
intuitive  form  for  managing  input  interaction  necessitated  placing  a  windowing 
frame,  around  each  displayed  video  source  for  direct  manipulation.  This  would 
give  the  look  and  feel  of  any  other  application  window  to  which  traditional 
computer  users  have  become  accustomed  (Figure  4). 
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Figure  4  Video  Window  Configuration  Enhancements 


3.1. 4.3  Architectural  Design 

Achieving  fluid  manipulation  of  live  data  windows  across  the  DataWall  display 
includes  the  following  elements: 

•  The  Fenestra  application  which  provides  transparent  management  of 
video  windows 

•  An  array  of  SuperView™  3000  systems  equal  to  the  number  of  display 
devices 

•  The  primary  DataWall  PC  for  communicating  and  exchanging  RS-232 
commands  with  the  multiple  SuperView™  systems 

•  Desired  external  video  sources  to  be  displayed  in  real  time  (can  include 
computer  and  low-res  video  types  such  as  laptops,  video  cameras,  DVD 
players  and  TV  tuner  signals) 

•  Distribution  amplifier  for  each  video  source  to  broadcast  the  same  signal 
to  each  screen 

The  initial  approach  of  displaying  sources  on  a  single  large  display  device  was  be 
to  map  and  attach  each  SuperView™  system  to  a  single  display  device  directly. 
Although  the  overall  display  appears  tiled,  the  manipulation  is  restricted  to  the 
corresponding  display  device.  The  implementation  for  allowing  Fenestra  involves 
daisy-chaining  each  video  processor  box  and  program  RS-232  commands  for 
upstream  and  downstream  communication  via  the  primary  DataWall  PC  driving 
the  display.  In  the  case  of  the  DataWall,  three  SuperViews™  were  utilized 
correlating  to  a  three  projector  DataWall  design  (Figure  5). 

The  three  video  outputs  from  the  DataWall  PC  are  each  connected  to  the 
background  input  channel  of  a  SuperView™,  and  subsequently  to  each  display 
device.  Unlike  other  controllers  whose  primary  purpose  involves  presenting 
multiple  computer  and  video  inputs,  the  organization  provides  the  familiarity  and 
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the  use  of  resources,  such  as  running  applications,  and  opening  files  resident  on 
the  DataWall  PC. 

The  current  implementation  allows  up  to  four  external  sources  for  window 
manipulation  on  the  DataWall  PC.  Each  external  video  source  is  connected  to  a 
distribution  amplifier,  so  that  it  may  be  input  to  each  SuperView™  unit.  The 
rationale  behind  this  approach  is  to  allow  each  complete  input  signal  to  be 
distributed  across  all  SuperViews™  in  order  to  be  displayed  anywhere  on  the 
tiled  display.  Depending  on  the  desired  location  for  displaying  the  information, 
segments  of  the  window  are  matched  with  the  appropriate  coordinates  to 
simulate  the  appearance  of  seamless  window  movement  across  all  projected 
areas. 


Figure  5  Fenestra  Architecture 

For  hiding  and  explicitly  allowing  only  the  desired  input  overlay  (such  as  the 
video  and  computer  videos)  to  pass  through  the  background  overlay,  a  technique 
called  chromakeying  is  utilized.  An  uncommonly  used  color,  such  as  bright 
fuchsia  has  been  chosen  for  the  current  implementation.  Only  against  fuchsia- 
filled  regions  in  the  background  overlay  are  computer  and  video  inputs 
programmed  to  appear.  The  rationale  behind  using  an  off  color  for  chromakeying 
lies  on  limiting  the  probability  of  allowing  any  artifact  with  the  same  hue  to  bleed 
unintentionally.  This  artifact  is  akin  to  weather  forecasters  who  have  accidental 
reflected  weather  maps  on  parts  of  their  clothing  corresponding  to  the  same 
chromakey. 

Each  fuchsia  colored  window  dragged  across  the  screen  has  a  corresponding 
input  source  commanded  to  intentionally  bleed  through  this  window,  and  thus 
appear  to  be  contained  inside  a  perimeter.  The  most  current  version  of  Fenestra 


13 


leverages  on  the  same  familiarity  and  look-and-feel  as  any  standard  Microsoft® 
Window. 

The  Fenestra  application  allows  all  components  to  work  synchronously,  and 
conducts  all  resources  to  achieve  the  required  wall  controller  management  tools. 
All  source  code  is  based  on  the  .NET  framework  utilizing  C#.  Numerous 
languages  and  packages  were  investigated  and  evaluated  to  achieve  the 
enhancement. 


3.1. 4.4  User’s  View 

A  key  feature  of  Fenestra’s  design  is  hiding  the  complexities  associated  with 
effective  video  window  management.  From  the  user’s  point  of  view,  a  simplified 
process  is  all  that  is  necessary  to  interact  with  windows.  Connecting  any 
computer  or  video  device,  launching  the  Fenestra  software,  and  simply  selecting 
any  of  the  external  sources  to  be  visible  comprise  the  steps  in  displaying  any  live 
window  on  a  tiled  display.  Communication  between  all  the  components  is 
performed  automatically.  The  abstractions  are  hidden,  and  the  user  needs  to  only 
run  the  program. 

From  the  user’s  perspective,  windows  are  moved  seamlessly  across  display  tiles 
without  any  notion  of  the  background  architecture,  incorporated  techniques,  and 
complexity  of  timing  parameters  and  layers  overlays.  All  window  inputs  can  be 
scaled,  minimized,  maximized  and  dragged  as  intuitively  as  any  standard 
Window  and  present  live  video  and  computer  signals  in  real  time  (Figure  6). 


High  Resolution 
Display  1 


_High  Resolution 
Displa>T2 


High  Resolution 


Figure  6  Fenestra  User’s  Perspective 
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3.1. 4.5  Fenestra  User  Interface  Description 

As  discussed  earlier,  Fenestra’s  interface  compared  to  the  SuperView™ 
manufacturer’s  software  is  significantly  more  user-friendly  and  transparent 
(Figure  7).  In  addition,  window  manipulation  is  performed  directly  on  the  video 
and  computer  input  across  tiles,  and  not  on  the  limited  real  estate  of  the  video 
window  controller  interface. 


Figure  7  Fenestra  Interface 


3.1. 4.6  Functions 

•  Direct  Window  Management  (Figure  8) 

o  The  primary  function  of  Fenestra  is  to  enable  direct  manipulation  of 
external  sources  as  intuitively  as  possible.  Each  input  is 
represented  as  a  standard  Microsoft®  Window  that  can  be 
dragged,  minimized,  maximized,  resized  and  dragged  anywhere 
across  all  tiles. 


Figure  8  Fenestra  Window  Management 
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•  Real-Time  Performance  of  Computer  and  Video  Inputs 

o  Display  and  removal  of  external  computers  or  video  sources  on  a 
high  resolution  display  is  easily  achieved  by  selecting  the 
appropriate  input. 

o  External  sources  to  be  displayed  are  hot-swappable.  Any  external 
input  can  be  displayed  without  restarting  the  program. 

•  Bordered  and  Borderless  Modes  (Figure  9) 

o  The  addition  and  removal  of  external  sources’  borders  create 
borderless  portals  that  still  maintain  the  ability  to  be  resizable  and 
movable  across  tiles.  The  objective  of  this  feature  was  to 
accommodate  multi-headed  machines  to  be  tiled  and  manipulated 
seamlessly  within  the  large  tiled  display. 


The  settings: 

-  3  Windows  turned  on 

-  Borders  option  unchecked 

Allows  piping  3-headed 
systems  to  be  displayed 
seamlessly  within  a  tiled 
display 


|  Fenestra 


Windows  Timing  Adjustments  Terminal  | 
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W  Window  1 
W  Window  2 
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Adjust  Attributes 

Window  1 

Window  2 

Window  3 


P  Borders 


Window  4 
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Figure  9  Borderless  Window  Mode 
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•  Simplified  Visual  and  Interactive  Timing  Adjustments  (Figure  10) 

o  The  Fenestra  occludes  the  numerical  complexities  and  technical 
jargon  involved  in  the  series  and  layers  of  overlays  to  achieve 
generating  the  most  accurate  timing  parameters  for  displaying 
external  sources  within  a  tiled  display  environment.  The  Timing 
Adjustment  panel  eases  configuring  the  timings  of  each  of  the  tiles 
within  the  background  display  (desktop),  and  the  external 
computers  and  video  sources  connected  to  the  SuperViews™. 


Screen 
timing 
intuitively 
maps  to 
each  tile  on 
the  display 


Each 
Window 
Timing 
intuitively 
maps  to 
all  inputs 
for  timing 


Screen  timing 
is  visually 
interactive  by 
pressing  the 
correct 
directional 
button,  with 
visual 
feedback 


llllJll-llliaillll.I.HL.II.I.U.imLIJ.UM  Xj 

X 
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Window  input 
timing  is 
visually 
interactive  by 
pressing  the 
correct 
directional 
button  with 
visual  feedback 


Figure  10  Fenestra  Tinning  Parameter  Adjustments 
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•  Accessible  RS-232  Terminal  (Figure  11) 

o  For  more  advanced  functions  and  commands  to  control  the 
SuperViews™,  and  to  cater  to  more  experienced  users,  a  RS-232 
terminal  was  developed.  The  terminal  also  serves  as  a  tool  for 
troubleshooting  the  system,  serving  as  an  all-else  fails  interface  to 
send  commands  manually  to  the  SuperViews™. 


BSl  Fenestra 


iBII^JSl  x| 


Windows  |  Timing  Adjustments  Terminal 


:unassign 
EC>  :assign 
Device  0  >  [select  0 
Device  0  >  CK  ON 

Device  0  >  CKC  522  1250  -270  175  522  1250 
Device  0  >  [select  1 


Send  Command 


Figure  11  RS-232  Terminal  Window 


•  Accessible  Individual  Preferences  for  Inputs  (Figure  12) 

o  Carefully  chosen  user  preferences  for  each  live  window  input  can 
be  configured.  The  panel  is  accessible  by  a  right  mouse  button  click 
on  any  of  the  windows,  or  by  selecting  the  window  number  on  the 
main  menu.  For  better  mapping  of  each  window  to  each  panel,  title 
bars  for  each  window  bears  the  same  window  label.  Changes  can 
be  performed  from  the  window  adjustment  panel  for  identifying  the 
proper  signal  source,  desired  aspect  ratio  of  the  signal,  and 
preferred  attributes  to  improve  acuity, 
o  Source  Type  adjustment  -  The  source  type  which  selects  the  type 
of  incoming  signal  whether  it’s  RGB  or  composite  input, 
o  Aspect  Ratio  adjustment  -  A  selection  of  commonly  standard  video 
and  computer  aspect  ratios  are  available,  4:3,  5:4,  and  16:9  (HDTV 
resolution) 

o  Acuity  adjustments  -  A  signal  processed  through  the  channels  can 
appear  differently  from  the  source  to  the  display  destination. 
Catering  to  these  conditions  brought  about  the  following  commonly 
adjusted  attributes:  brightness,  contrast,  sharpness  and  hue. 
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Figure  12  Video  Window  Adjustment  Preferences 


•  Window  Identify  Confirmation  (Figure  13) 

o  Large  tiled  displays  in  general  require  multiple  windows  which  fills 
the  work  space  area.  Differentiating  which  window  needs  current 
adjustment,  when  multiple  windows  are  open  is  beneficial. 


Identify  Window  is  the 
transparent  way  of 
confirming  the  current 
selected  window  in  a  large 
area  display.  Numerical 
Information  appears  on  the 
window  overlay 


Figure  13  Identify  Window 
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•  Accessible  and  User  Friendly  Help  File 

o  Although  the  Fenestra  interface  is  designed  to  be  user-centric,  a 
help  file  is  also  available  to  ensure  proper  setup,  and  procedure  for 
operation.  Information  such  as  a  general  overview,  a  step-by-step 
general  setup,  necessary  components,  and  an  anticipated  listing  of 
question  in  the  “how  do  I”  section  complete  the  help  file.  Most  of  the 
information  on  the  help  file  are  accompanied  by  guided  graphics  to 
further  illustrate  the  necessary  information  to  enable  easy  user 
operation 


3.1. 4.7  Testing  and  Evaluation 

The  challenges  posed  by  the  limitations  of  the  SuperView™,  and  a  need  for 
effective  integration  into  the  DataWall  system  resulted  in  Fenestra.  It  is  an 
architecture  composed  of  both  software  and  hardware  components  which  fulfills 
the  objective  of  providing  an  intuitive  video  windowing  system  to  display  multiple 
video  and  computer  inputs  on  a  tiled  large-screen  display.  The  implementation 
overcomes  the  restrictions  of  the  SuperView™  while  delivering  a  capability  for 
operators  to  display,  position,  and  scale  multiple  sources  across  a  multi-headed 
Windows  environment  in  real  time.  It  has  been  designed  to  hide  the  complexities 
of  the  components  and  procedures  from  the  user. 

The  Fenestra  system  was  integrated  in  the  Built-In  DataWall.  Positive  feedback 
regarding  the  usability  was  received  during  these  trials.  Demonstrations  include 
running  intensive  graphics  and  applications  smoothly.  It  has  provided  an  avenue 
for  the  DataWall  to  be  a  more  widely  used  display  resource  for  in-house  staff  and 
visitors  to  connect  multiple  laptops,  and  hot  swap  external  inputs  with  ease  to 
accomplish  their  tasks. 


3.2  Wireless  Interaction 


3.2.1  Rationale 

Most  large  high-resolution  displays  available  in  the  present  market  are 
manufactured  as  passive  systems.  These  systems  typically  only  allow  the  user  to 
view  data,  and  provide  little  means  for  engagement  and  interaction.  A  factor  that 
separates  the  DataWall  and  goes  beyond  its  capability  to  provide  an  extensive 
global  view  of  the  information  space,  is  its  ability  to  function  as  an  active  and 
interactive  work  space  and  not  merely  as  a  summary  device  for  information 
presentation. 
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Collaborators  face  the  challenge  of  effectively  managing  massive  amounts  of 
data  displayed  on  a  large  screen.  The  use  of  input  devices  such  as  conventional 
mice  and  keyboards  is  a  cumbersome  and  limiting  approach,  which  tethers  user 
operations  to  a  single  location.  The  primary  mode  for  managing  and  controlling 
information  on  the  DataWall  screen  is  through  wireless  input  devices.  The 
DataWall  has  capitalized  on  current  speech  recognition  technology  via  wireless 
microphones  and  cordless  telephones.  Camera-tracked  laser  pointers  offer  a 
more  natural  and  direct  way  of  interacting  with  a  large  screen  with  mouse-like 
functionality.  For  users  who  would  like  to  leverage  familiar  input  devices,  and  yet 
would  appreciate  a  more  ubiquitous  interaction,  the  DataWall  also  supports 
conventional  mice  and  keyboards  as  alternatives. 


3.2.2  Challenges 

Operating  wirelessly  has  clear  advantages.  However,  for  security  reasons  use  of 
wireless  devices  is  restricted  within  Department  of  Defense  (DoD)  installations 
and  strictly  forbidden  in  some  cases  including  the  use  of  COTS  wireless 
keyboards.  Any  wireless  device  to  be  integrated  needs  to  comply  with  security 
policies.  Devices  need  to  scale  well  for  very  large  screens  so  that  cursor 
movement  is  effortless  and  efficient.  Devices  also  need  to  be  accurate  and 
intuitive  to  allow  operators  to  use  the  display  effectively.  Intuitiveness  is  the  most 
challenging  by  far. 

The  ADII  program  investigated  the  use  of  several  wireless  input  devices  with  a 
focus  on  the  development  of  a  speech  recognition  system  and  a  camera-tracked 
laser  pointer  which  are  discussed  here. 


3.2.3  Continuous  Speech  Recognition 


3.2.3.1  Introduction 

Incorporated  in  the  earlier  versions  of  the  DataWall  is  commercially  available 
voice-recognition  software  developed  by  NUANCE  Communications.  This 
program,  together  with  a  wireless  microphone  or  a  cordless  telephone,  enables  a 
user  to  experience  a  fast,  hands-free  utility  for  display  interaction.  A  user  issues 
verbal  commands,  which  the  software  captures,  analyzes  and  executes  by 
matching  these  instructions  against  a  predefined  grammar  set.  Such  a  voice- 
driven  mode  of  input  enables  the  user  to  perform  common  desktop  operations 
such  as  manipulating  windows,  navigating  menus,  or  executing  customized 
instructions  inherent  in  particular  applications. 

The  NUANCE  software  is  also  speaker-independent.  The  software  needs  no 
training  from  a  specific  user  to  recognize  instructions,  and  works  well  with  accent 
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and  voice  variability.  This  proves  to  be  an  essential  feature  in  collaborative 
environments  like  C2  situations  where  control  over  the  display  often  needs  to  be 
switched  from  one  user  to  another  or  multiple  people  frequently  engaging  with 
applications  simultaneously. 

The  most  current  speech  recognition  system  that  was  integrated  into  the 
DataWall  environment  used  a  novel  approach  by  utilizing  a  Dialogic  telephony 
card,  a  Skutch  telephone  line  simulator,  a  cordless  telephone,  and  the  Nuance 
software.  Using  a  cordless  telephone  in  lieu  of  a  cordless  microphone  had  the 
advantage  of  freeing  up  the  sound  card  for  other  duties  that  would  otherwise  be 
dedicated  to  the  microphone.  It  also  allows  multiple  telephones,  thus  multiple 
speakers  to  utilize  the  speech  system  simultaneously. 


3.2.3.2  Testing  and  Results 

Although  utilizing  continuous  speech  recognition  presents  many  advantages  and 
a  variety  of  ways  for  automating  tasks  verbally,  only  a  comparatively  small 
amount  of  users  would  exercise  this  mode  of  interaction  with  the  DataWall.  The 
primary  reason  for  its  limited  acceptance  is  due  to  the  fair  amount  of  learning  and 
familiarity  needed  with  the  defined  grammar  set  to  become  proficient  with  a 
voice-driven  tool.  In  addition,  some  users  have  reported  some  uneasiness  about 
wearing  additional  external  gear  such  as  a  headset. 

There  is  a  learning  curve  to  develop  grammar  sets  that  make  this  enhanced 
capability  effective,  but  it  is  no  steeper  than  learning  any  typical  software 
package.  In  C2  situations  where  interaction  with  various  applications  is 
imperative,  the  use  of  voice  recognition  can  be  quite  useful.  It  is  common 
practice  to  have  an  operator  responsible  for  the  timely  switching  and  coordination 
of  applications  being  displayed.  Voice  recognition  and  interaction  allows  the 
presenter  to  take  control  of  this  task  with  simple  utterances. 

It  has  been  very  challenging  to  develop  grammar  sets  that  are  generic  enough  to 
use  with  any  application.  Inevitably,  to  effectively  voice-enable  an  application 
requires  a  custom  grammar  set  for  each  application.  It  is  for  this  reason  speech 
input  was  all  but  been  abandoned  as  a  mode  of  interaction  in  the  DataWall 
project.  However,  there  is  still  interest  in  furthering  its  development  when  coupled 
to  other  input  devices. 


3.2.4  Laser  Pointer  Interaction 

To  provide  a  wireless  mouse-like  mode  of  input,  a  camera-tracked  red  laser 
pointer  system  was  developed.  Three  video  cameras  are  positioned  behind  the 
screen  above  each  video  projector.  The  live  video  from  the  three  cameras  is 
processed,  and  when  a  laser  dot  is  detected  on  the  screen,  the  cursor  is 
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positioned  at  its  location.  In  addition  to  cursor  positioning,  the  system  includes 
provisions  for  mouse  button  clicks  to  allow  all  the  functionality  of  a  conventional 
mouse  including:  dragging/dropping  windows,  resizing  windows,  and  interacting 
with  graphical  user  interface  (GUI)  widgets  such  as  buttons  and  scroll  bars.  An 
important  aspect  of  the  laser  pointer  input  device  is  that  it  is  essentially 
application  independent.  However,  since  the  Interactive  DataWall  runs  on  a  PC 
platform  under  Microsoft®  Windows,  the  application  must  be  Windows  compliant. 
The  only  other  requirement  is  it  must  have  a  GUI. 

Initial  in-house  development  of  the  laser  pointer  interaction  system  pre-dates  this 
in-house  program,  but  contributed  to  the  current  system’s  evolution.  The  very  first 
laser  pointer  tracking  system  consisted  of  three  PCs  equipped  with  video  capture 
cards.  The  next-generation  laser  tracking  system  consisted  of  a  custom 
hardware  solution  that  was  also  developed  prior  to  this  in-house  program. 
However,  it  was  used  extensively  in  several  Interactive  DataWall 
implementations  developed  under  this  program,  the  details  of  which  are 
described  here.  The  most  current  implementation  of  the  laser  pointer  tracking 
system  utilizes  a  frame  grabber  installed  in  the  DataWall  PC  that  also  drives  the 
display.  What  follows  are  descriptions  of  these  various  laser  tracking 
subsystems. 


3.2.4.1  Custom  Hardware  and  Software 

3.2.4.1.1  Technique  and  System  Description 

The  very  first  in-house  developed  laser  pointer  tracking  implementation  that  pre¬ 
dates  this  in-house  program  consisted  of  three  PCs  equipped  with  video  capture 
cards.  These  computers  provided  a  frame  by  frame  screen  capture  for  each 
video  camera  positioned  behind  the  screen.  The  frames  were  analyzed  via  in- 
house  developed  software  on  the  PCs  and  subsequently  transmitted  to  the 
display  computer  for  cursor  positioning.  Although  quite  effective,  the  process 
suffered  several  limitations.  First,  the  system  could  not  operate  in  real  time.  The 
approach  required  that  the  camera  and  computer  complete  a  frame  before 
analysis  could  begin.  The  delay  penalty,  therefore,  was  never  less  than  the  time 
required  to  complete  a  given  frame.  Second,  both  resolution  and  update  rate 
were  constrained  by  the  processing  power  available  at  the  time.  A  minimally 
acceptable  320  x  240  pixel  image  at  15  frames/second  would  fully  consume  the 
resources  of  a  high-end  PC  at  the  time.  Lastly,  the  cost/benefit  ratio  was  difficult 
to  justify  as  even  a  minimal  system  would  cost  several  thousand  dollars,  and 
multiple  PCs  were  required  in  these  multi-camera  implementations  (i.e.  one  PC 
per  video  camera/display  tile). 

To  address  the  limitations  of  the  original  PC/Video  Capture  Card  implementation, 
the  ADII  team  invented  and  patented  (Pat.  No.  6,377,242)  specialized  hardware 
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to  track  the  laser  pointer.  It  functioned  in  near  real  time,  with  readily  expandable 
resolution,  and  at  a  small  fraction  of  the  cost. 

In  this  implementation  the  three  video  cameras  positioned  behind  the  screen  are 
equipped  with  red  filters.  The  cameras’  views  of  the  screen  are  dark  fields  until 
the  laser  dot  came  into  a  camera’s  field  of  view.  The  live  video  from  the  three 
cameras  is  processed  by  the  custom  hardware  and  the  data  is  subsequently  sent 
to  the  DataWall  computer  for  proper  positioning  of  the  cursor  (Figure  14). 


Figure  14  AFRL/IF’s  Custom  Laser  Pointer  Tracking  Subsystem  Architecture 


The  device  combines  a  microcontroller,  video  processing  logic,  a  pair  of 
counters,  and  real  time  control  logic,  which  together  track  the  pointer  image.  The 
process  involves  synchronizing  the  counters  to  follow  the  camera  video.  The 
cameras’  views  of  the  screen  are  dark  fields  until  the  laser  dot  comes  into  a 
camera’s  field  of  view.  At  the  point  in  time  when  the  camera  “sees”  the  pointer, 
the  counters  will  contain  values  representative  of  the  pointer’s  position  on  the 
display  surface.  These  values  are  then  passed  on  to  the  display  driver  in  near 
real  time  via  serial  cable  to  position  the  cursor. 

In  order  to  evaluate  the  effectiveness  of  the  device,  a  side  by  side  comparison 
between  it  and  a  200MHz  Pentium  PC  implementation  was  conducted.  The 
exercise  confirmed  several  significant  advantages  of  the  new  system. 

First,  speed :  The  device  can  easily  produce  a  detection  each  time  the  video 
scans  the  pointer  while  the  original  PC  system  could  only  consistently  maintain  a 
detection  rate  of  20-25%  of  the  scan.  Second,  timeliness :  The  device  begins  to 
report  a  detection  almost  immediately  after  the  video  signal  “sees”  the  pointer 
while  the  PC  system  must  always  complete  a  full  frame  before  its  analysis  can 
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even  begin.  Third,  resolution:  In  order  to  sustain  a  10-15Hz-update  rate,  the  PC 
version  was  limited  to  a  resolution  of  320  x  240  pixels.  The  device,  is  capable  of 
512  x  480  pixels,  and  can  readily  be  increased  to  any  practical  video  resolution 
without  penalty.  Fourth,  and  finally  cost:  The  prototype  single-camera  device  was 
assembled  at  a  cost  of  less  than  $150  while  the  cost  of  the  PC  system  was  in 
excess  of  $2500.  The  single-camera  tracking  device  or  PC  was  required  for  each 
display  tile.  The  device  pictured  above  could  process  all  three  camera  inputs, 
and  several  were  built  for  a  total  cost  of  about  $450  each. 


3.2.4.1.2  Testing  and  Results 

The  custom  laser  tracking  hardware  implementation,  first  prototyped  in  1998, 
was  a  significant  improvement  to  its  predecessor.  It  was  not  without  its  flaws 
however.  Varying  lighting  conditions  made  calibration  and  continued  operation  of 
the  laser  tracking  problematic  particularly  as  the  DataWall  development  ventured 
into  portable  systems  where  lighting  was  difficult  to  control.  It  was  extremely 
difficult  to  get  the  camera  iris  adjusted  properly  to  ensure  detection  of  the  laser 
dot  on  the  screen  while  avoiding  false  detections  of  ambient  light.  The  accuracy 
of  the  tracking  was  also  less  than  optimal  in  that  the  cursor  was  always 
positioned  several  pixels  from  the  actual  laser  dot  on  the  screen.  Experienced 
operators  were  able  to  adjust  for  the  error,  but  novice  users  had  great  difficulty 
using  the  device.  Even  with  its  flaws  it  provided  a  novel,  scalable,  and  the 
beginnings  of  a  very  intuitive  method  for  wireless  interaction  on  a  large  display.  It 
was  used  extensively  in  several  Interactive  DataWall  implementations  until  the 
development  of  the  current  laser  tracking  system  that  utilizes  a  COTS  frame 
grabber  was  first  prototyped  during  the  Summer  of  2002. 


3.2.4.2  COTS  Frame  Grabber  and  Custom  Software 
3. 2.4. 2.1  Technique 

The  technique  to  track  the  laser  pointer  has  come  full  circle  back  to  a  PC  with  a 
frame  grabber  approach.  PC  hardware  advances  and  associated  dramatic 
performance  improvements  over  the  last  several  years  has  also  allowed  the 
image  processing  to  be  executed  by  the  central  DataWall  computer  rather  than 
an  independent  system.  The  most  current  implementation  of  the  laser  tracking 
system  utilizes  a  commercially  available  high  performance  multi-channel  frame 
grabber  card  to  capture  video  feeds  from  three  black  and  white  cameras  that  are 
synchronized  with  a  black  burst  generator  (Figure  15).  An  individual  camera 
captures  all  live  images  mapped  to  each  display  tile.  Analog  signals  are  saved 
and  converted  to  a  format  intelligible  for  digital  processing  by  software  that 
utilizes  a  COTS  imaging  library.  Custom  software  developed  with  the  University 
of  Alabama  (contract  no.  FA8750-04-C-0067),  together  with  the  video  frame 
input,  executes  a  very  reliable  algorithm  for  accurately  detecting  the  locations 
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and  precise  movements  of  processed  laser  dots  and  strokes  within  the  display 
space  to  manifest  effective  laser  pointer  interaction. 


I 


Figure  15  Frame  Grabber  Laser  Pointer  Tracking  Subsystem  Architecture 


Initial  versions  of  the  camera-tracked  laser  pointer  software  were  designed  for 
single  user  interaction,  thus  it  detected  and  supported  only  one  cursor  at  a  single 
point  in  time.  The  current  algorithm  enables  multiple  laser  dots  to  be  active,  and 
multiple  users  to  simultaneously  interact  with  the  DataWall  display  provided  they 
each  work  on  different  applications. 

Improvements  for  tracking  multiple  lasers  dots  have  been  applied  as  well. 
Procedures  to  allocate  detection  and  simultaneous  filtration  of  multiple  laser  dots 
have  been  implemented.  The  initial  algorithm  for  detecting  laser  dots  employed 
scanning  line  per  line  from  top  to  bottom  looking  for  the  first  pixel  of  a  laser  dot  as 
the  frames  are  captured  by  the  camera.  Predefined  bounding  boxes  are  utilized 
to  search  for  multiple  laser  dots  on  the  viewed  camera  frame.  This  approach  may 
produce  inaccurate  results  in  scenarios  where  laser  dots  are  in  very  close 
proximity  of  each  other  as  a  result  of  overlapped  search  boxes.  Currently,  a  more 
refined  approach  creates  bounding  boxes  when  entire  dots  are  identified  first, 
preceding  the  recognition  of  the  exact  bounding  box.  In  this  manner,  as  long  as 
two  bounding  boxes  do  not  intersect,  it  easily  distinguishes  individual  laser  dots. 

Improvements  to  the  algorithm  also  include  a  divide-and-conquer  screen 
masking  technique  for  cameras  to  ignore  any  events  outside  the  DataWall 
display  area.  The  initial  implementation  examined  each  pixel  within  the  camera 
image  for  inclusion  or  exclusion  to  the  mask.  The  redesigned  method  considers 
only  every  four  corners  of  the  camera  image  plane.  If  all  of  the  corners  are  within 
the  mask,  then  the  quadrant  is  included  in  the  mask.  This  technique  immensely 
accelerates  the  masking  process  as  compared  to  validating  each  of  the  camera 
image’s  307200  pixels.  The  applied  masking  process  effectively  prevented  false 
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positives,  and  focuses  the  processed  area  to  be  within  the  proximity  of  the 
DataWall  screen. 

One  of  the  limitations  imposed  by  the  Standard  Microsoft®  Windows  OS  is  its 
dedicated  single  cursor  framework.  The  enhanced  customized  software  extends 
the  VNC-based  environment  for  allowing  simultaneous  multi-laser  pointer 
interaction.  A  Regular  VNC  remote  session  allows  any  single  PC  (viewer)  to  have 
full  operation  of  a  networked  system  (server).  The  algorithm  extends  the  remote 
session  framework  and  incorporates  the  notion  of  borrowing  external  cursors 
from  multiple  remote  systems.  Running  the  customized  software,  at  most 
accounts  for  x+1,  the  number  of  cursors  simultaneously  utilized.  The  number  x 
maps  to  the  count  of  external  application  systems  (servers)  that  are  connected 
via  the  network,  and  have  running  programs  pushed  to  the  DataWall  (viewer) 
screen  area  for  interaction. 

Accommodating  the  specified  multiple  cursor  platform,  it  follows  that  at  most  x+1 
laser  pointers  can  also  be  independently  used  on  the  display  area  at  the  same 
time.  Implementation  of  the  laser  detection  is  through  a  stroke-based  multiple 
cursor  approach.  Strokes  are  formed  by  detecting  and  examining  each  laser  dot, 
verifying  whether  it  falls  on  the  category  of  the  start  of  an  active  stroke,  or  the 
continuation  of  one.  Active  strokes  have  estimated  next-positions  based  on  the 
attributes  of  location  and  speed.  Each  dot  is  validated  as  a  continuation  of  a 
stroke  if  it  falls  within  the  next-position  tolerances  of  the  available  active  strokes, 
or  whether  they  signal  the  beginning  of  a  new  one.  A  stroke  ends  if  no  dots  fall 
within  the  range  of  the  estimated  next-position  of  a  stroke.  Calculations  are 
performed  whether  active  strokes  together  with  currently  detected  laser  dots  are 
determined  to  be  inside  or  outside  the  pushed  applications.  Strokes  that  fall 
within  these  windowed  applications  have  the  ability  to  independently  maneuver 
the  cursor  for  that  window.  Although  strokes  per  window  can  manifest  multiple 
cursors  within  the  DataWall  area,  only  one  stroke  or  cursor  can  claim  ownership 
of  an  externally  originating  application  at  a  single  point  in  time. 

The  VNC  framework  has  its  limitations  when  conducted  on  a  large  tiled  display. 
Core  to  its  implementation  requires  transferring  the  entire  screen  of  a  remote 
computer  to  be  operated  as  an  application  on  the  DataWall.  The  process  of 
mapping  each  pixel  of  the  remote  screen  to  the  DataWall  is  proportionally 
bounded  by  a  combination  of  factors:  the  network  bandwidth,  the  amount  of 
pixels  distributed  along  the  network,  and  the  amount  of  activity  attributed  to  the 
application.  Videos  and  highly  intensive  graphics  are  more  prone  to  elicit  sluggish 
refreshes  while  being  run. 

A  driver-based  approach  is  utilized  to  address  refresh  problems.  The  primary 
advantage  of  this  technique  is  based  on  intercepting  and  transferring  only 
commands  for  drawing  instead  of  pixels  from  the  originating  remote  system  to 
the  DataWall  PC.  Instead  of  a  high  volume  mapping  of  remote  to  local  pixels 
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being  transferred  to  the  display,  the  device  driver  approach  requires  only  drawing 
commands  to  be  transmitted  from  the  source.  The  load  of  data  required  to  move 
across  the  network  is  greatly  diminished  to  produce  remote  applications  running 
more  fluidly  and  closer  to  the  actual  display  time.  Though  the  driver-based 
technique  enhances  the  display  rate  and  is  almost  fully  functional  it  is  still  under 
development. 


3. 2. 4. 2. 2  Camera  Alignment  and  Calibration 

Critical  to  accurate  laser  tracking  is  properly  positioning  the  video  cameras.  The 
cameras  must  view  the  image  area  of  its  respective  projector  and  little  else. 
Slight  overlap  coverage  among  the  cameras  is  acceptable  because  the  laser  pen 
tracking  software  will  handle  this.  However,  the  goal  is  to  maximize  the  use  of 
each  camera’s  sensor  in  order  to  provide  the  laser  tracking  software  with  the 
highest  resolution  image  possible.  The  cameras  are  equipped  with  zoom  lenses 
and  mechanically  adjustable  mounts  to  accomplish  this  manually.  Once  the 
cameras  have  been  aligned,  a  map  must  be  established  between  the  camera 
space  and  the  display  space.  The  process  of  establishing  the  map  is  referred  to 
as  automatic  camera  calibration.  To  establish  the  map,  the  system  first  takes  a 
picture  of  a  blank  screen,  then  of  a  matrix  of  reference  white  dots  on  a  black 
background.  The  centroids  of  those  white  dots  on  the  camera  image  are 
identified.  Finally  a  map  between  the  centroids  in  the  left  side  figure  and  those  in 
the  right  side  figure  are  established  (Figure  16).  To  map  a  point  in  the  camera 
space  to  a  point  in  the  display  space,  a  bilinear  interpolation  algorithm  is  used  [1]. 
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Figure  16  Mapping  Between  Camera  Space  and  Display  Space 

Automatic  camera  calibration  has  many  advantages.  A  map  of  size  4x5,  7x9,  and 
10x13  can  be  successfully  created  in  just  a  matter  of  seconds.  Early  versions  of 
the  laser  tracking  calibration  required  a  very  tedious  manual  process  of 
positioning  the  laser  dot  at  each  calibration  point  which  would  take  several 
minutes  even  with  a  small  number  of  points.  It  was  also  very  difficult  to  accurately 
position  the  laser  pointer  at  all  the  calibration  points.  With  automatic  calibration 
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even  larger  map  sizes  can  be  used  to  improve  laser  tracking  accuracy  with  no 
impact  on  the  calibration  process  that  would  otherwise  be  even  more  tedious 
using  the  manual  process.  Images  taken  by  a  camera  with  a  wide  angle  lens 
have  a  severe  radial  distortion.  To  improve  the  accuracy  of  the  mapping,  more 
calibration  points  can  be  used  and  a  piecewise  bilinear  interpolation  with  a  larger 
number  of  pieces  can  be  performed. 

Enhancements  to  the  camera  calibration  design  focuses  on  safeguarding 
operations  to  assure  precise  laser  pointer  functionality.  One  of  the  tools  added 
includes  better  automated  software  that  calibrates  cameras  in  proper  order,  and 
disables  continuation  of  the  process  when  anomalies  are  detected.  Such 
anomalies  can  include  unintended  illumination  of  objects  in  the  room  or 
reflections  on  the  screen  that  could  be  mistaken  for  calibration  points;  or 
incomplete  camera  capture  of  the  display  space  due  to  incorrect  camera 
positioning.  Simple  correction  adjustments  by  the  operator  trigger  the  completion 
of  the  configuration.  In  addition,  utilities  are  available  for  advanced  users  to 
select  alternative  arrangements,  and  other  adaptive  filtering  processes. 


3.2.4.2.3  Simulating  Mouse  Button  Operations 

To  make  a  laser  pointer  work  like  a  mouse,  an  interface  was  designed  to  perform 
mouse  button  operations  such  as  click,  double-click,  drag,  etc.  A  commercially 
available  laser  pointer  has  limited  functions.  Basically,  it  has  one  single  button  to 
turn  the  laser  on  and  off.  However,  the  laser  pointer  button  operations  map 
naturally  to  the  mouse  button  operations.  To  simulate  multiple  mouse  button 
operations  a  mouse  resource  window  (MRW)  is  used. 

Version  1.0  of  the  MRW  has  4  buttons  (Figure  17). 

The  top  left  one  is  used  to  simulate  the  left  mouse 
button,  and  the  top  right  one  to  simulate  the  right 
mouse  button.  If  the  left  button  mode  is  selected,  the 
laser  button  will  simulate  the  left  mouse  button:  a 
push  operation  on  the  laser  pointer  button  will 
generate  a  left  mouse  button  down  event,  and  a 
release  operation  on  the  laser  pointer  will  send  a  left 
mouse  button  up  event.  In  the  same  principle,  if  the 
right  button  mode  is  selected,  the  laser  button  will 
simulate  the  right  mouse  button.  In  principle,  a  left 
mouse  button  click  operation  can  be  performed  by 
pushing  and  releasing  the  laser  pointer  button. 

However,  the  buttons  of  some  commercially 
available  laser  pointers  cannot  react  fast  enough  to 
simulate  the  click  operation,  and  the  laser  dot  will  move  when  the  button  is 
pressed,  especially  if  it  is  pressed  at  a  distance.  For  this  reason,  two  more 
buttons  were  added  to  the  MRW.  The  bottom-left  button  is  used  to  simulate  the 


Figure  17  Mouse  Resource 
Window  Version  1.0 
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single  click  operation  on  the  left  mouse  button,  and  the  bottom-right  button  is  to 
simulate  the  double  click  operation.  When  no  button  is  selected  in  the  MRW,  the 
movement  of  the  laser  dot  corresponds  to  the  cursor  movement  in  the  display. 

Mouse  button  operations  are  a  sequence  of  mixed  left  button  and  right  button 
operations.  A  laser  pen  can  simulate  any  sequence  of  mouse  button  operations 
with  the  help  of  the  MRW.  However,  it  is  inconvenient  to  switch  back  and  forth 
between  the  main  application  window  and  the  MRW.  Any  sequence  of  operations 
can  be  divided  into  several  subsequences,  each  of  which  is  dominated  by 
operations  of  a  single  button.  For  example,  a  mouse  button  operation  sequence 
can  mainly  consists  of  left  mouse  button  operations  preceded  by  a  few  right 
button  operations  for  menu  selection.  To  reduce  the  number  of  switches  between 
the  main  application  window  and  the  MRW,  a  temporal  mode  in  addition  to  the 
permanent  mode  was  integrated.  When  the  laser  pointer  clicks  on  the  right  side 
of  a  button  in  the  MRW,  one  operation  of  that  mouse  button  will  be  reserved. 
Multiple  reservations  can  be  made  by  clicking  the  right  side  of  the  button  multiple 
times.  When  the  laser  pointer  clicks  on  the  left  side  of  a  button  in  the  MRW,  it 
sets  up  the  permanent  mode.  Figure  16  illustrates  the  permanent  left  button 
mode  with  one  right  button  operation  reserved.  The  laser  pointer  will  simulate 
one  right  button  operation  followed  by  left  button  operations  until  the  mode 
changes.  The  sequence  can  be  used  to  bring  up  a  menu,  make  a  selection,  and 
then  continue  with  left  button  operations. 

The  current  implementation,  version  2.0,  provides  the  same  indispensable 
mouse  functions  with  a  more  clear  representation  of  both  available  and  currently 
selected  functions  (Figure  18).  All  MRW  buttons  are  permanently  labeled  with 
text  as  to  their  function  rather  than  when  they’re  selected  as  in  the  previous 
version  of  the  MRW.  The  color  of  a  button  is  more  intuitively  shifted  to  indicate  it 
has  been  selected. 


Figure  18  Current  Mouse  Resource  Window  Version  2.0  and  Task  Box 
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Additionally,  a  Task  Box  Interface  accompanies  each  MRW  on  the  display 
workspace.  The  Task  Box  grants  participants,  via  the  laser  pointer,  to  scroll 
through  the  list  of  all  open  programs  currently  running  on  any  remotely  connected 
computer  and  to  select  a  program  to  display  on  the  DataWall  screen.  Moreover, 
available  files,  and  commonly  used  programs  can  also  be  launched  by  selecting 
icons  among  those  available  on  the  scrollable  horizontal  collection. 

Multiple  remotely  connected  systems  can  display  applications  to  the  DataWall, 
thus  allowing  simultaneous  laser  pointer  interaction.  Each  application  shared  has 
a  matching  mouse  resource  window  and  task  box  forming  a  set  specific  for  that 
application.  A  distinct  color  is  assigned  to  intuitively  define  and  group  the 
ownership  of  one  set,  and  to  provide  a  solid  visual  mapping  during  multiple 
concurrent  laser  pointers  interacting  on  the  display  space  (Figure  19). 


Figure  19  Color  Feedback  for  Multiple  Simultaneous  Laser  Pointer  Interaction 
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4  Portable  Interactive  Data  Walls 


4.1  Rationale 

As  early  as  1998  AFRL/IF  recognized  the  utility  of  developing  a  field  deployable 
version  of  the  Interactive  DataWall  to  get  the  display  and  HCI  technology  out  of 
the  laboratory  and  into  the  hands  of  potential  users  for  evaluation  to  guide  further 
research  and  development.  Participating  in  various  military  exercises  and 
conferences  across  the  country  would  expose  many  more  potential  customers 
that  might  not  otherwise  have  the  opportunity  to  visit  our  laboratory.  It  could  also 
provide  a  display  system  to  showcase  other  AFRL/IF  developed  software  that 
could  benefit  from  the  DataWall’s  display  and  HCI  capabilities. 


4.2  Challenges 

Designing  and  implementing  a  large-screen  display  system  based  on  Interactive 
DataWall  technology  that  could  be  taken  off  site  and  used  in  a  variety  of 
environments  was  extremely  challenging.  A  laboratory  is  a  very  controlled 
environment.  The  DataWall  relied  heavily  on  this  for  correct  operation  including 
controlled  lighting  conditions  for  the  projection  room  to  optimize  projected  images 
and  the  camera-based  laser  tracking.  The  unit  had  to  be  portable  and 
deployment/set-up  had  to  be  very  efficient  to  ensure  timely  operation.  First 
attempts  were  designed  around  the  assumption  that  the  DataWall  developers 
would  support  the  deployment  providing  the  necessary  expertise  to  get  the 
system  operational.  The  ultimate  goal  was  to  develop  a  system  that  could  be 
taken  anywhere,  required  minimal  tear-down/set-up,  and  tuned  by  inexperienced 
operators. 


4.3  Deployable  Interactive  DataWall  (DID)  1998-1999 

The  first  deployable  version  of  the  Interactive  DataWall  was  housed  in  an 
extensively  modified  Air  Force  S-530  A/G  Standard  Rigid  Walled  shelter,  with  its 
own  Tactical  Generator  Set  and  Environmental  Control  Unit  (Figure  20).  It  was 
completely  self  sufficient  since  it  generated  its  own  power  and  provided  the 
necessary  cooling  or  heating  as  the  case  may  be  for  its  electronic  components. 
Three  LCD-based  projectors  with  short-throw  lenses  were  used.  In  this 
configuration,  each  projector  displayed  1024  x  768  pixels  for  a  total  display 
resolution  of  3072  x  768  across  a  screen  area  9'  x  2%'.  At  the  time  this  was  state- 
of-the-art  in  lightweight  LCD-based  projector  technology.  Although  the  system 
was  developed  during  the  1998-1999  time  period,  before  this  in-house  program 
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began  it  was  the  first  implementation  of  a  portable  DataWall  and  is  described 
here  to  understand  the  evolution  of  these  portable  systems. 


Figure  20  Deployable  Interactive  DataWall 


4.4  Portable  Interactive  DataWall  (PIDW)  1999-2000 

The  Deployable  Interactive  DataWall  proved  to  be  a  valuable  asset  for  off-site 
demonstration  support,  but  required  a  15K  forklift  and  flatbed  truck  to  transport  it. 
A  Portable  Interactive  DataWall  (PIDW)  designed  to  be  lightweight,  easily 
disassembled,  transported  in  light-duty  trucks,  and  reassembled  very  quickly  was 
then  developed.  It  would  be  designed  so  that  once  assembled  it  could  be  very 
easily  rolled  to  another  location  by  two  people.  It  would  be  PC-based  and  need  to 
fit  through  a  standard  3'  doorway.  Although  a  common  practice  for  shortening  the 
footprint  of  a  rear  projection  display  is  to  incorporate  folded  optics  (i.e.  mirrors),  a 
conscious  decision  was  made  to  avoid  this.  Mirrors  add  complexity  to  the 
alignment  procedure,  add  weight,  and  are  fragile  if  an  optical  quality  glass  is 
used. 

A  prototype  system  was  completed  May  2000  with  a  footprint  of  9%'W  x  5'H  x 
2%'D  when  folded  and  9%'W  x  6%'H  x  4%'D  when  extended  and  operational.  It 
was  also  a  3  x  1  LCD-based  projector  configuration.  Each  high-resolution 
projector  displays  1280  x  1024  pixels  for  a  total  display  resolution  of  3840  x  1024 
pixels  across  a  screen  area  9'  x  2%'.  Projectors  and  cameras  are  a  straight  throw 
to  the  screen  with  wide-angle  lenses  to  accommodate  the  extremely  short-throw 
requirements.  For  deployment  the  lock  pins  on  the  hinged  screen  frame  support 
arms  are  removed  to  fold  the  screen  down  in  front  of  the  unit.  The  screen  is  a 
flexible  snap-on  material  than  can  also  be  easily  removed  and  rolled  up  for 
deployment.  The  DataWall  PC,  audio  amplifier,  video  overlay  hardware,  and 
video  equipment  such  as  DVD  players  or  video  cassette  recorders  are  mounted 
in  an  integrated  rack  system. 
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4.5  Modular  Portable  Interactive  DataWall  (PIDW  v2)  2000-2001 


The  prototype  PIDW  was  deployed  several  times  to  support  off  site 
demonstrations,  conferences,  and  military  exercises  that  provided  valuable 
insights  into  improvements.  A  common  problem  identified  was  the  inability  to  get 
the  PIDW  into  certain  areas  because  of  its  length  of  9'.  Negotiating  it  into  a  room 
from  a  narrow  corridor  was  impossible.  The  shipping  container  for  the  unit  was 
also  extremely  large  and  heavy  which  still  required  a  forklift  for  deployment,  albeit 
smaller  than  what  was  required  to  deploy  the  DID. 

The  next  version  of  the  PIDW  was  designed  to  be  modular  and  could  be 
disassembled  into  sections  to  allow  greater  flexibility  for  set-up  in  tight  spaces 
and  to  allow  transport  in  smaller  containers  that  would  fit  on  a  standard  463L 
cargo  pallet;  an  important  characteristic  necessary  for  overseas  deployment  by 
military  transport  (Figure  21).  Shock-mounts  were  added  to  the  equipment  rack 
to  allow  a  majority  of  the  equipment  to  remain  in  place  when  the  system  was 
transported.  This  significantly  reduced  the  number  of  shipping  containers  and 
greatly  simplified  disassembly  and  reassembly  because  all  connected  cables 
could  remain  intact.  Two  prototypes  were  completed  January  2001. 

Another  area  that  needed  improvement  was  the  projector  mounts.  Although  they 
did  provide  6  degrees  of  adjustment  (x,  y,  z,  roll,  pitch,  and  yaw)  to  achieve 
image  alignment,  the  lack  of  fine  tune  adjustment  for  x,  z,  and  yaw  made  the 
process  very  tedious.  It  required  manually  sliding  the  projectors  on  slotted  plates 
for  each  adjustment  that  were  very  difficult  to  move  with  the  precision  required.  A 
new  projector  mount  was  designed  that  included  a  threaded  adjuster  that  could 
be  turned  at  small  increments  to  reposition  each  axis  more  smoothly  and 
precisely.  A  prototype  mount  was  fabricated,  but  problems  with  machining  the 
adjustment  fittings  to  the  necessary  tolerances  prevented  the  design  from  being 
integrated. 


Figure  21  Modular  Portable  Interactive  DataWall 
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4.6  Collapsible  Interactive  DataWall  (CIDW) 


4.6.1  CIDW  Version  1  Prototype  2001-2002 

The  latest  version  of  the  DataWall  under  development  takes  portability  to  a  new 
level.  A  primary  objective  throughout  the  portable  DataWall  development  has 
been  not  only  to  provide  a  minimal  operational  footprint,  but  also  a  minimal 
transport  footprint.  Using  commercially  available  reconfigurable  T-slotted 
aluminum  extrusions  and  fasteners,  a  collapsible  frame  with  minimal  disassembly 
was  developed.  It  is  designed  to  allow  the  entire  frame  to  collapse  such  that  the 
frame  will  occupy  a  much  smaller  footprint  when  folded  for  transport  and  fits  in  a 
single,  more  manageable  container  (Figure  22). 


Side  View 


Front  View 


Collapsed 


Figure  22  Collapsible  Interactive  DataWall  Version  1 


4.6.2  Advantages  of  Material 

The  80/20  material  has  a  number  of  advantages  over  conventional  materials 
previously  used  in  earlier  version  of  the  Portable  Interactive  DataWalls.  Some  of 
the  accessories  available  include  lockable  sliding  bearings  that  allow  sections  of 
the  frame  to  collapse  and  extend  rather  than  be  disassembled  and  reassembled. 
This  makes  deployment  and  set-up  effortless  and  requires  no  tools.  Using  the 
COTS  material  allows  quick  and  efficient  fabrication  of  the  DataWall  frames  that 
have  previously  been  custom  fabricated  with  riveted  tubular  aluminum.  It  also 
simplifies  design  changes  and  can  be  modified  easily.  The  material  has  been 
evaluated  very  favorably  on  the  following  criteria:  cost,  weight,  ease  of  assembly, 
consistence  of  assembly  on  deployments,  and  structural  rigidity.  It  is  now  the 
material  of  choice  for  all  our  current  and  ongoing  display  frame  projects. 
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4.6.3  Prototype  Design  Issues 


The  first  prototype  completed  July  2002  had  a  number  of  issues  that  were 
identified.  For  cost  reasons  the  first  design  tried  to  keep  the  use  of  bearing 
systems  for  the  folding  parts  to  a  minimum.  The  design  relied  heavily  on  hinges 
and  lock  pins  to  extend  and  collapse  the  frame.  However,  the  lock  pin 
mechanism  had  too  much  play  in  the  connection  and  did  not  give  the  structural 
rigidity  required.  There  were  also  issues  with  the  upper  horizontal  screen 
supports  from  which  the  screen  was  suspended.  The  weight  of  the  screen  frame 
caused  the  support  arms  to  sag  because  they  were  not  adequately  supported 
along  their  span.  The  original  design  also  required  removal  and  repositioning  of 
certain  support  extrusions  to  collapse  the  unit.  The  goal  was  to  keep  assembly 
for  deployments  to  a  minimum. 


4.6.4  CIDW  Version  2  2002-2003 

The  next  version  of  the  frame  was  a  significantly  more  simplified  and  elegant 
design  that  used  nylon  slide  bearings  at  all  collapsing  points.  The  additional  cost 
far  outweighed  the  benefits  they  provide.  Each  folding/extending  point  can  now 
be  very  securely  locked  into  place  in  either  position.  The  base  of  the  unit  when  in 
operational  mode  is  also  much  smaller.  The  projector  platform  now  relies  solely 
on  the  strength  of  the  locked  bearings  rather  than  vertical  support  beams  that 
previously  connected  the  top  and  bottom  sections  of  the  frame.  The  upper  screen 
support  arms  were  redesigned  to  telescope  rather  than  fold  and  additional 
vertical  supports  were  added  that  corrected  the  sagging  problem  identified  in  the 
prototype’s  design.  The  screen  frame  was  also  redesigned  to  be  more  rigid  by 
replacing  the  folding  hinge  mechanism  with  a  sturdy  butt  joint  that  connects  the 
left  and  right  screen  frame  sections  together.  Although  it  now  requires  the  screen 
frame  to  be  disassembled  into  two  pieces  for  deployment  it  is  significantly  more 
secure  once  assembled.  Version  2  with  the  upgrades  described  was  completed 
January  2003  (Figure  23). 
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Figure  23  Collapsible  Interactive  DataWall  Version  2 

As  COTS  projector  technology  improved  new  products  were  integrated  into  the 
system.  To  avoid  a  major  redesign  of  the  frame,  investigation  into  a  projector 
upgrade  that  had  similar  characteristics  was  conducted.  Most  critical  were  the 
projector  footprint,  resolution,  and  throw  distance  to  image  size  ratio.  The  only 
choice  available  was  a  new  product  from  NEC,  model  GT2150.  It  met  all  the 
requirements  with  the  added  benefits  of  being  brighter  and  with  a  unique 
motorized  vertical  and  horizontal  lens  shift  capability.  The  lens  shift  provided  an 
invaluable  capability  to  electro-mechanically  shift  the  images  left-to-right  and  up- 
and-down,  dramatically  simplifying  and  improving  the  alignment  procedure  that 
previously  had  to  be  accomplished  mechanically  via  the  rudimentary  projector 
mounts. 
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4.6.5  Projector  Mount  Redesign  and  Integration  2003-2004 

Although  the  lens  shift  capability  of  the  new  projectors  provided  a  much  needed 
improvement  to  the  alignment  procedure  it  only  provided  adjustment  for  the  x  and 
y  positions  and  did  not  provide  the  fine  tune  adjustment  that  was  desired.  A  new 
6  Degrees  of  Freedom  (DOF)  projector  mount  was  developed  that  incorporated 
some  elements  from  the  previously  unsuccessful  design,  but  took  advantage  of 
the  frame  extrusion  T-slots  to  provide  a  channel  for  sliding  linear  bearing 
mechanisms  for  the  x  (left/right)  and  z  (forward/backward)  adjustment.  Each  can 
be  adjusted  by  a  threaded  rod  and  knob  that  can  be  turned  for  precise 
repositioning  of  the  projector.  The  y  position  (height),  pitch,  and  roll  can  be 
adjusted  by  a  series  of  knobs  positioned  at  each  of  the  four  corners  of  the 
projector  mount.  Yaw  can  be  adjusted  by  rotating  the  top  of  the  projector  mount 
that  pivots  at  a  fixed  position  in  the  center  (Figure  24). 

The  new  projector  mount  was  a  substantial  improvement  and  was  able  to  take 
advantage  of  the  80/20  characteristics  for  a  more  smooth  and  precise  positioning 
mechanism  that  was  unsuccessfully  attempted  previously.  Off  the  shelf  parts 
were  used  without  requiring  precision  machining  of  any  components.  The  locking 
mechanism  was  also  an  improvement  that  requires  no  tools  and  does  not  affect 
the  projector  position  when  locked  into  place.  Both  were  issues  with  the  previous 
projector  mounts.  The  new  projector  mount  assembly  design  and  integration 
were  completed  January  2004. 


Figure  24  CIDW  6  DOF  Projector  Mount 
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4.6.6  Integrated  Shipping  Container  Design  and  Integration  2004 

To  further  improve  portability  of  the  CIDW  system,  a  shipping  container  that  was 
also  more  portable  was  developed.  Custom  fabricated  containers  made  of 
plywood  had  been  used  previously  and  were  very  heavy  and  difficult  to  move. 
Containers  for  the  Portable  DataWalls  have  drastically  decreased  in  size  through 
it  evolution  but  even  in  its  now  very  compact  form  the  CIDW  alone  is  still  very 
heavy,  compounded  with  the  weight  of  a  still  sizable  wooden  container. 

Initial  experiments  to  add  casters  and  a  ramp  to  the  container,  to  roll  the 
collapsed  CIDW  inside  provided  some  benefit,  but  there  was  also  the  issue  of 
storage  space  for  the  container  when  it  was  deployed.  First  attempts  to  design  a 
container  that  could  be  disassembled  for  better  storage  were  unsuccessful 
because  it  comprised  its  structural  integrity.  It  was  also  still  fabricated  from  heavy 
plywood  which  made  disassembly  and  reassembly  very  difficult. 

Manufacturers  that  offered  custom  containers  made  from  lightweight  plastic  were 
contacted.  One  material  that  was  identified  had  a  unique  laminated  cellular 
construction  that  was  claimed  to  be  comparable  in  strength  to  ABS-clad  plywood. 
It  also  offered  the  ability  to  construct  a  container  of  unconventional  size  that  was 
not  possible  with  molded  plastic  at  a  reasonable  cost.  After  researching  the  types 
of  containers  these  manufacturers  provide,  lead  to  a  unique  approach  of 
integrating  the  container  into  the  CIDW  frame,  rather  than  rolling  it  into  an 
independent  container.  Using  a  clamshell  container  design  the  base  of  the 
container  was  fitted  with  casters.  The  casters  on  the  CIDW  were  removed  and 
the  unit  placed  in  the  container  base.  When  the  CIDW  is  fully  collapsed  the  clam 
shell  sides  are  positioned  on  the  base,  clasped  together  and  the  base  to  fully 
contain  the  unit.  The  lightweight  plastic  is  extremely  easy  to  lift  into  place  by  a 
single  person.  The  container  was  fabricated  and  integrated  into  the  CIDW 
September  2004. 


4.6.7  Touch  Screen  Collapsible  Interactive  DataWall  2004-2005 

A  frequent  question  from  visitors  to  the  ADII  research  facility  and  DataWall  users 
is  whether  a  touch  screen  interface  could  be  integrated.  Until  recently  there  were 
many  limitations  to  touch  screen  technology  that  made  integration  into  the 
DataWall  difficult  and  also  compromised  image  quality.  A  common  COTS 
approach  is  to  use  touch  sensitive  film  that  lays  over  the  display  surface.  It  is 
never  a  completely  transparent  material  which  subsequently  degrades  the  image 
quality  of  the  display.  They  were  also  not  scalable  to  very  large  displays. 
Perimeter  infrared  sensors  were  also  an  early  approach  which  lacked  the 
accuracy  and  resolution  needed,  and  like  the  touch  sensitive  films  were  also  not 
very  scalable.  They  were  best  suited  for  smaller  flat  panel  displays. 
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COTS  manufacturers  have  since  developed  touch  screen  technology  using 
camera-based  sensors.  Although  still  marketed  for  smaller  flat  panel  displays  and 
single  screen  rear  projection  display  systems,  the  approach  has  great  potential 
for  scalability  to  much  larger  multi-projector  display  systems. 

Development  of  a  touch  screen  capability  for  the  CIDW  is  currently  under 
development.  Because  of  the  extensive  modifications  required,  a  completely  new 
CIDW  was  designed  and  fabricated  for  the  touch  screen  research  so  that  our 
current  system  would  remain  operational  for  demonstration  support.  Starting  with 
the  current  design  as  a  baseline  a  rigid  screen  is  being  integrated,  and 
modifications  were  done  to  the  frame  to  incorporate  new  projectors  and 
accommodate  the  additional  weight  of  the  screen. 


4.6.7.1  Rigid  Screen  Investigation 

The  CIDW  was  chosen  as  an  experimental  platform  for  investigating  the 
feasibility  of  a  touch  screen.  The  first  challenge  was  to  integrate  a  rigid  screen 
that  was  still  portable  and  could  be  disassembled,  but  also  rigid  enough  for  touch 
interaction  when  assembled. 

Three  rigid  screens  each  36"W  x  29"H  that  met  the  requirements  for  throw 
distance  and  best  estimate  for  thickness  were  purchased.  A  perimeter  screen 
frame  to  tile  the  three  pieces  together  made  of  the  same  80/20  material  from 
which  the  CIDW  frame  was  constructed  was  also  purchased.  The  screen  frame 
was  also  sectional  for  portability.  A  number  of  problems  were  immediately 
apparent  during  the  initial  testing  phase.  First  and  foremost,  although  the  screen 
was  %"  thick  it  was  not  rigid  enough  for  the  application.  Because  the  screen 
material  was  only  captured  around  the  perimeter  of  the  entire  3x1  tiled  screen, 
the  individual  screen  tiles  flexed  too  much  at  the  seams  when  touched.  The 
screen  was  also  a  2-element  material  that  was  difficult  to  keep  tightly  together 
with  this  type  of  screen  frame.  This  also  contributed  to  the  flexing  and  introduced 
some  ghosting  to  the  projected  images. 

In  light  of  this  it  was  determined  either  a  thicker  or  denser,  more  rigid,  single 
element  screen  with  a  custom  screen  frame  would  be  necessary.  Discussions 
have  started  with  a  number  of  COTS  screen  designers  and  vendors  to  procure  a 
screen  that  meets  our  specifications,  but  no  design  has  been  finalized. 


4.6.7.2  Projector  Identification 

As  projector  technology  improves  COTS  models  are  discontinued  or  replaced 
with  models  of  slightly  different  specifications.  Although  the  portable  DataWalls 
have  always  been  designed  to  allow  hardware  upgrades  with  minimal 
modifications  they  are  still  bound  by  projector  footprint  and  throw  distance. 
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Selection  of  COTS  projectors  that  meet  our  requirements  for  footprint,  resolution, 
throw  distance  to  image  size  ratio,  and  power  has  been  extremely  limited. 

The  trend  in  projector  technology  has  been  away  from  the  SXGA  resolution  of 
1280  x  1024  pixels  toward  SXGA+  of  1400  x  1050  and  high  definition  (HD)  of 
1920  x  1080.  Although  there  is  a  desire  to  eventually  use  the  HD  format 
projector,  model  selection  is  very  limited  and  currently  geared  toward  the  home 
theater  market.  These  applications  are  typically  front  projection  with  throw 
distance  to  image  size  ratios  greater  than  1:1,  designed  for  longer  throw 
distances  much  greater  than  3’.  So,  finding  a  model  with  a  short  throw  lens 
option  that  accomplishes  a  throw  ratio  less  than  or  equal  to  1:1  for  rear  projection 
has  been  even  more  limited. 

An  SXGA+  DLP®  projector  was  selected  from  Christie  Digital,  model  Matrix 
3000,  based  on  availability  and  the  closest  in  specifications  to  previous  projectors 
used  for  the  CIDW.  They  have  the  benefit  of  motorized  horizontal  and  vertical 
lens  shift  similar  to  the  projector  they  replace.  They  are  also  specifically  designed 
for  multi-projector  applications  with  more  consistent  color  and  brightness. 


4.6.7.3  CIDW  Frame  Redesign 

The  Christie  Matrix  3000  projector’s  footprint  and  offset  lens  required  some 
modifications  to  the  CIDW  integrated  projector  mounts.  A  slightly  shorter  throw 
ratio  also  required  the  screen  to  be  mounted  a  few  inches  closer  to  the 
projectors. 

The  most  significant  change  to  the  design  was  the  screen  support  structure  to 
accommodate  the  additional  weight  of  the  new  rigid  screen  and  screen  frame 
that  is  estimated  to  weigh  in  excess  of  100  lbs.  The  telescoping  upper  screen 
support  arms  were  redesigned  with  an  integrated  metal  rail  bearing  to  replace  the 
nylon  bearing  system  in  the  previous  design.  The  lower  screen  support  arms 
were  also  redesigned  with  a  similar  telescoping  rail  bearing  system  to  replace  the 
simple  locking  pivot  joints  used  previously.  The  vertical  bearing  system  that 
raises  and  lowers  the  entire  screen  support  assembly  were  also  redesigned  with 
a  rail  bearing  system  to  replace  the  nylon  bearings.  In  addition  hydraulic  pistons 
with  a  motorized  pump  raise  and  lower  the  screen  assembly  at  the  touch  of  a 
button  that  previously  had  to  be  lifted  and  lowered  manually  (Figure  25).  The  rail 
bearings  are  each  rated  for  a  dynamic  load  of  2810  lbs  which  far  exceeds  the 
entire  screen  assembly  weight  and  should  prove  more  than  adequate  for  precise 
extension  and  retraction  of  the  rigid  screen.  Maximum  operating  temperature  is 
175°F  with  an  accuracy  of  0.001"  of  running  parallelism.  Although  the  rail  bearing 
load  capacity  was  overkill,  they  were  the  narrowest  available  rails  that  would 
mate  directly  to  the  extrusion  and  provide  sufficient  surface  area  for  solid 
mounting. 
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Figure  25  Touch  Screen  CIDW  Design  Concept 


4.6.7.4  Touch  Sensor  System 

The  software  and  hardware  that  will  accomplish  the  touch  sensing  will  be 
contracted  out  to  a  commercial  touch  screen  developer.  It  is  anticipated  that 
current  off-the-shelf  technology  can  be  leveraged  and  scaled  to  a  much  larger 
tiled  display  than  what  is  currently  available. 


4.6.8  CIDW  Marketing 

Although  the  CIDW  continues  to  be  improved  the  most  current  system  is 
extremely  well  designed  with  an  abundance  of  functionality  not  currently 
available  commercially.  Working  with  a  local  contractor  a  CIDW  information 
packet  was  created  in  the  event  that  if  a  potential  customer  was  interested  in 
acquiring  one,  a  detailed  description  and  pricing  would  be  available.  The 
contractor  is  poised  to  take  over  the  production  and  support  for  any  future 
systems  sold. 


5  Technology  Transitions 

The  Interactive  DataWall  in  its  various  stages  of  development  provided  several 
technology  transition  opportunities,  particularly  the  portable  versions.  Several 
systems  were  transitioned  off-site  to  environments  that  ranged  from  experimental 
facilities  to  operational  theater.  The  details  of  these  transitions  including  lessons 
learned  and  current  status  are  discussed  below. 
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5.1  51st  Fighter  Wing,  Osan  Air  Base,  Korea 


A  Portable  Interactive  DataWall  (PIDW)  was  installed  26  November  2001  at  the 
51st  Fighter  Wing  at  Osan  Air  Base,  Korea  to  support  the  Restoration  of 
Operations  (RestOps)  Advanced  Concept  Technology  Demonstration  (ACTD) 
funded  through  the  Defense  Threat  Reduction  Agency  (DTRA).  RestOps 
simulates  the  recovery  of  operations  from  a  chemical  or  biological  attack.  The 
PIDW  was  part  of  an  effort  to  move  the  recovery  center's  command  and  control 
from  one  based  on  voice  communications  to  one  based  on  sharing  of  data 
electronically.  Previously,  field  teams  assessed  battle  damage,  collected  and 
analyzed  potential  contamination  after  an  attack  and  then  relayed  the  results  via 
radios  or  in  person  to  the  Wing  Operations  Center  (WOC).  Staff  at  the  WOC 
would  utilize  large  maps  with  Plexiglas  overlays  and  grease  pencils  to  show  the 
status  of  the  base  after  the  attack.  RestOps  provides  an  interactive  C2  planning 
tool  that  automates  the  flow,  analysis  and  display  of  information  from  the  field  to 
the  WOC  and,  therefore,  very  effectively  demonstrated  the  DataWall's  ability  to 
improve  collaboration. 

Although  they  did  not  have  a  need  for  a  portable  display  that  would  be  moved 
frequently,  the  PIDW’s  minimal  operational  footprint  and  self-contained  design 
were  the  characteristics  most  attractive  to  the  RestOps  system  developers. 
Elaborate  facility  modifications  such  as  a  rear  projection  screen  installation  and 
projection  room  construction  were  unnecessary  to  integrate  the  display  system 
into  their  facility.  The  system  was  assembled,  loaded  with  software  and  was  fully 
operational  soon  after  delivery. 


5.1.1  Lessons  Learned 

The  PIDW  proved  very  useful  for  the  preliminary  phase  of  RestOps  particularly 
its  minimal  impact  on  the  facility  in  which  it  was  installed.  It  is  important  to  point 
out  that  because  RestOps  is  an  application  that  would  run  immediately  after  an 
attack,  personnel  using  it  would  be  wearing  fully  encapsulating  chemical  warfare 
defense  (CWD)  gear.  This  early  PIDW  employed  the  wireless  interactive 
capabilities  available  in  today’s  model.  However,  early  implementation  of  the 
laser  interaction  was  not  very  accurate  and  personnel  had  difficulty  working  with 
it  because  of  the  CWD.  Speech  recognition  was  also  not  a  mode  of  interaction 
suitable  for  this  type  of  application.  Furthermore,  even  if  the  application  utilizes 
hot  keys  that  could  make  use  of  the  speech  recognition  it  would  be  nearly 
impossible  because  of  the  CWD  mask  that  is  worn. 

Another  area  of  concern  for  the  developers  and  users  of  RestOps  is  the  color 
balance  and  hot  spots,  which  can  be  attributed  to  the  inherent  differences  that 
exist  from  projector  to  projector  even  when  they  are  the  same  model.  Although 
RestOps  is  not  a  mapping  application  in  its  entirety,  it  relies  on  a  map  to  show 
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the  progress  of  field  checks  across  an  installation.  The  color  imbalance  and  hot 
spotting  could  in  some  instances  be  a  distraction  to  users. 

Difficulty  for  inexperienced  users  to  effectively  align  the  projectors  was  realized 
well  before  RestOps,  but  had  yet  to  be  addressed.  RestOps  did  reinforce  the 
necessity  to  develop  a  more  effective  alignment  mechanism. 


Figure  26  RestOps  DataWall  at  Osan  AB  Korea 


5.1.2  Impact  and  Current  Status 

RestOps  and  use  of  the  PIDW  by  other  organizations  has  resulted  in  significant 
improvements  to  the  data  interaction  as  well  as  set-up  and  portability.  For 
instance,  the  importance  of  improving  the  laser  pointer  interaction  was  realized. 
The  summer  of  2002,  began  the  development  of  the  laser  pointer  interaction 
software  that  utilized  a  COTS  frame  grabber.  The  result  was  far  more  accurate 
and  more  easily  calibrated  than  its  custom  hardware  predecessor.  The  goal  is  to 
allow  any  user  to  quickly  calibrate  the  software  and  begin  interacting  with  the 
data.  In  places  like  Osan  AB  where  personnel  rotate  every  15  months,  it’s 
important  to  provide  an  interface  which  is  intuitive. 

COTS  and  university  developed  hardware  and  software  are  available  to 
accomplish  more  seamless  tiled  images  with  more  accurate  color  balance. 
However,  because  they  use  edge  blending  and  image  warping,  a  conscious 
decision  was  made  not  to  implement  them  to  preserve  the  full  resolution  of  the 
display.  Proper  image  alignment  is  still  required  even  in  instances  where  these 
techniques  are  implemented.  Effort  was  focused  on  developing  a  more  effective 
fine-tune  alignment  mechanism  that  was  recently  integrated  into  the  most  current 
CIDWs.  Significant  advancements  in  COTS  projector  technology  over  the  last 
few  years  have  seen  drastic  improvements  in  color  and  brightness  consistency. 
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Although  these  improvements  have  solved  some  of  the  issues,  there  are  still 
plans  to  implement  some  level  of  automatic  color  balance  adjustment  in  future 
systems. 

RestOps  has  transitioned  into  a  larger  situational  awareness  C2  tool  for 
Commanders.  Although  it  has  not  been  approved  for  fielding,  many  instances  of 
the  application  have  been  distributed  throughout  the  military.  Developers  insist 
laser  interaction  is  not  an  effective  means  of  working  with  the  data  because  it 
slows  users  down.  The  argument  is  that  a  majority  of  the  users  have  grown  up 
playing  PC  games  and  have  mastered  the  use  of  the  mouse,  thus  being  able  to 
navigate  through  the  application  at  a  faster  rate  than  with  the  laser  pointer. 
Collaboration  with  the  developers  to  find  an  effective  way  to  interact  with  the  data 
will  continue,  especially  when  the  application  must  run  after  an  attack  and 
personnel  are  wearing  cumbersome  CDW  gear.  The  system  remained 
operational  until  the  next  phase  of  the  RestOps  program  which  required  a  much 
larger  display  than  this  portable  system  provided.  However,  the  Portable 
DataWall  proved  very  useful  for  the  preliminary  phase  and  valuable  insights  from 
users  were  instrumental  in  its  continuing  evolution. 


5.2  Army  10th  Mountain  Division ,  Fort  Drum  NY 

The  10th  Mountain  Division  is  a  light  infantry  unit  dating  back  to  World  War  II. 
Although  it  specializes  in  mountain  fighting,  the  unit  has  evolved  into  one  capable 
of  fighting  in  any  terrain.  But  as  the  name  implies,  it  is  light  infantry  with  no  heavy 
fighting  vehicles.  This  in  turn  allows  them  to  deploy  quickly  and  tackle  a  wide 
variety  of  contingencies. 

On  3  December  2001,  AFRL/IF  was  contacted  by  the  10th  Mountain  Division 
regarding  the  immediate  deployment  of  one  of  the  AFRL  Portable  Interactive 
DataWalls  (PIDWs)  in  support  of  the  war  on  terrorism.  Personnel  of  the  10th 
Mountain  Division  were  familiar  with  the  PIDW’s  capabilities  as  a  result  of 
participation  in  a  number  of  field  tests  performed  jointly  at  Ft.  Drum,  NY  over  the 
past  four  years.  Although  designed  for  indoor  use,  the  lightweight  design  of  the 
PIDW  was  critical  to  the  Light  Infantry  10th  Mountain  Division’s  mission.  Their 
Chief  of  Staff  was  introduced  to  the  technology  during  these  exercises,  and  in 
responding  to  a  need  for  immediate  operational  deployment  of  his  unit  requested 
a  PIDW  be  prepared  and  immediately  shipped  to  Ft.  Drum.  The  DataWall  was  to 
accompany  his  unit  to  an  operational  theater  in  Afghanistan. 

An  available  DataWall  was  immediately  transported  from  AFRL/IF’s  Collaborative 
Simulation  Technology  &  Applications  Branch  at  Wright-Patterson  AFB,  OH  to 
Rome,  NY.  It  was  reassembled  by  the  ADII  team.  Photographs  were  taken  and 
an  assembly/user’s  manual  was  developed.  The  latest  incremental  design 
changes  were  integrated  and  tested.  New  shipping  containers  were  designed 
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and  fabricated.  The  DataWall,  assembly  instruction  manual,  a  field  spares  kit  and 
the  tools  needed  for  assembly  were  delivered  to  the  palletizing  area  at  Ft.  Drum 
on  4  December  2001  (28  hours  after  the  request).  The  following  week  the 
operators  that  were  scheduled  to  follow  the  equipment  to  the  field  came  to  Rome 
for  eight  hours  of  training.  At  that  time  they  learned  how  to  assemble,  align  and 
operate  the  equipment.  The  user’s  manual  was  used  during  the  training  session 
and  modified  based  on  trainee  feedback.  The  final  version  of  the  manual  was 
delivered  in  electronic  form  the  following  day.  This  PIDW  was  used  in  theater  for 
nearly  nine  months  as  a  C2  display  for  daily  update  briefings  from  the  various 
groups  within  the  Division  as  well  as  to  display  live  feeds  from  Intelligence, 
Surveillance,  and  Reconnaissance  assets.  An  additional  DataWall  was  requested 
with  a  three  month  lead  time  for  delivery.  The  2nd  system  was  delivered  on  28 
February  2002  (a  day  earlier  than  promised)  and  was  used  in  the  Division  Main 
HQ  Emergency  Operations  Center  (EOC)  at  Ft.  Drum. 


5.2.1  Lessons  Learned 

The  system  that  was  deployed  to  operational  theater  returned  to  AFRL/IF  March 
2003  and  provided  some  valuable  insight  into  further  improvements.  The  shock- 
mount  rack  was  not  integrated  at  the  time  these  systems  were  delivered  which 
now  greatly  simplifies  the  assembly  process  and  drastically  reduces  the  number 
of  shipping  containers  required  for  deployment.  The  PIDW  was  never  designed 
to  withstand  harsh  field  conditions,  which  is  how  it  was  used  while  deployed  to 
Afghanistan.  Since  the  COTS  electronic  components  have  standard  rack-mount 
chassis  with  standard  ventilation  slots  and  fans,  exposure  to  dust  posed  a 
substantial  problem.  No  equipment  failed,  but  when  AFRL/IF  had  the  system 
returned  to  our  laboratory  for  testing,  evaluation,  and  minor  frame  repairs,  all 
equipment  showed  evidence  of  substantial  exposure  to  dust.  The  projectors, 
which  are  not  designed  to  operate  continuously,  sustained  the  most  exposure  to 
dust  and  at  times  operated  erratically.  All  internals  were  professionally  cleaned 
for  safety  and  to  ensure  proper  operation.  Components  designed  for  harsh 
environments  are  currently  being  evaluated  and  methods  to  reduce  exposure  are 
being  considered  for  future  rugged  DataWall  frame  designs. 

Another  area  of  interest  for  this  type  of  field  use  of  the  DataWall  is  the  rapid 
deployment  and  set  up  of  the  system.  Minimizing  the  amount  of  disassembly, 
deployable  footprint,  and  reassembly  are  paramount.  Achieving  seamless 
boundaries  without  loss  of  resolution  is  also  important.  However,  in  situations 
where  rapid  deployment  and  setup  is  critical,  employing  a  blending  and 
alignment  algorithm  could  speed  up  the  process  and  allow  operators  to 
concentrate  on  other  tasks. 

The  DataWall  is  well  suited  for  small  collaborative  cells  and  the  way  the  10th 
Mountain  Division  (Figure  27)  employs  it  illustrates  this  point  best.  It  is  used  to 
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present  update  briefings  from  the  various  groups  within  the  Division  as  well  as 
displaying  live  feeds  from  Intelligence,  Surveillance,  and  Reconnaissance  (ISR) 
assets. 

Unfortunately  it  is  still  difficult  to  break  most  operators  from  the  common  mindset 
of  seeing  the  DataWall  as  simply  a  set  of  three  independent  video  projectors  to 
display  three  sources  of  information  at  a  time.  For  many,  how  a  meta-desktop 
can  be  effectively  used  for  data  management  is  a  difficult  concept  to  grasp.  Using 
the  three  projectors  with  three  independent  sources  is  still  the  most  common 
mode  of  operation.  AFRL/IF  continues  to  work  with  them  so  that  they  can  truly 
benefit  from  what  the  DataWall  can  provide  and  in  return  provide  field  experience 
feedback  to  guide  our  future  research. 


Figure  27  Portable  DataWalls  at  Ft  Drum  NY 


5.2.2  Impact  and  Current  Status 

Even  before  the  PIDW  redeployed  from  Afghanistan,  a  shock-mount  rack  system 
had  already  been  incorporated  which  reduced  its  transportation  footprint  by  75 
percent.  Considerable  improvements  were  made  in  set-up  requirements  which 
have  reduced  set-up  time  from  about  four  hours  to  one  hour  or  less.  A  rugged  PC 
that  could  withstand  extreme  temperatures  and  dusty  environments  is  currently 
under  investigation. 

The  relationship  with  Ft  Drum  personnel  continued  to  ensure  operation  and 
maintenance  procedures  were  current.  The  system  that  was  deployed  overseas 
and  returned  from  duty  in  March  2003  was  repaired  and  the  most  current  system 
upgrades  integrated.  After  which  the  system  was  returned  to  Ft  Drum  to  resume 
its  duties.  The  status  of  both  systems  owned  by  the  10th  Mountain  Division  is 
unknown  at  this  time  because  of  recent  and  continuing  personnel  deployments. 
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5.3  Electronic  Systems  Center,  Hanscom  AFB  MA 

In  May  2003  the  latest  version  of  the  Collapsible  Interactive  DataWall  (CIDW) 
was  installed  at  the  Electronics  Systems  Center  (ESC)  at  Hanscom  AFB  MA  to 
support  a  joint  Global  Concept  of  Operations  Synchronization  (GCS)  effort 
among  the  Air  Force  Research  Laboratory,  Air  Combat  Command,  Air  Mobility 
Command,  and  ESC.  The  objective  of  GCS  is  to  improve  information  sharing  and 
interoperability  between  Combat  Air  Forces  (CAF)  and  Mobility  Air  Forces  (MAF) 
mission  planning  and  execution  systems  for  improved  velocity,  efficiency,  safety, 
and  mission  success.  The  CIDW  is  a  key  system  in  the  architecture  established 
for  the  demonstration  because  of  its  ability  to  bring  distributed  C2  applications 
into  a  single,  interactive  large  display.  Some  of  these  applications  include  the 
Global  Air  Mobility  Advanced  Technology  (GAMAT)  as  well  as  the  Integrated 
Management  Technology  (IMT).  As  a  result  of  the  success  of  the  preliminary 
experimentation,  the  system  was  purchased  by  ESC  July  2003. 


5.3.1  Lessons  Learned 

The  CIDW  has  played  a  central  role  to  GCS  because  it  brought  new  capabilities 
and  efficiencies  in  data  processing,  flow,  and  visualization.  It  also  provided  that 
common  situational  awareness  environment  where  decision-makers  can  quickly 
identify  problems  and  be  proactive  in  solving  potential  ones.  In  concert  with 
previous  users  of  the  PIDW  and  CIDW,  there  have  been  some  concerns  about 
the  interaction  with  the  data  while  using  the  laser  pointer.  Feedback  provided  by 
the  users  point  to  the  difficulty  in  correctly  aligning  the  cameras.  Although  the 
calibration  routine  and  implementation  of  the  laser  interaction  is  now  very 
accurate,  proper  alignment  of  the  cameras  is  essential. 


Figure  28  Collapsible  Interactive  DataWall  at  ESC 


Portability  and  transportation  have  also  been  mentioned  as  problems.  GCS  has 
been  demonstrated  to  various  audiences  in  places  that  range  from  the  Pentagon 
to  large  convention  halls.  The  users  have  experienced  difficulty  in  dealing  with 
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the  weight  and  size  especially  when  large  trucks  and  forklifts  are  not  available. 
When  packaged  for  transportation,  the  CIDW  weighs  approximately  1100 
pounds.  Set-up  continues  to  be  a  key  concern  for  users.  A  significant  amount  of 
visual  systems  technology  and  computing  expertise  have  been  cited  as 
requirements  to  set  it  up  correctly. 


5.3.2  Impact  and  Current  Status 

Although  manual  alignment  of  projectors  is  a  time-consuming  process,  there  is 
no  loss  of  resolution  which  is  a  major  advantage;  especially  with  certain 
applications.  The  recent  upgrade  to  the  CIDW  projector  mounts  that  provides  six 
degrees  of  adjustment  enables  a  user  to  manually  and  precisely  align  the 
projectors  in  significantly  less  time. 

A  solution  to  aligning  the  cameras  employed  for  the  laser  pointer  interaction  has 
not  been  fully  realized.  The  initial  approach  has  been  improvements  to  the 
camera  calibration  process  that  is  significantly  more  simplified  and  less  critical  of 
misaligned  camera  positions  to  a  certain  degree.  Ultimately,  an  automatic 
process  where  the  typical  user  does  not  have  a  direct  influence  on  the 
optimization  of  the  camera  position  needs  to  be  developed. 

The  system  at  ESC  has  been  upgraded  with  the  most  current  improvements 
including  the  new  projector  mounting  and  alignment  system,  and  the  lightweight 
integrated  shipping  container. 


5.4  Northeast  Air  Defense  Sector 

In  the  Spring  of  2002  personnel  from  the  Air  National  Guard’s  Northeast  Air 
Defense  Sector  (NEADS)  approached  the  ADII  team  for  advice  on  upgrades  to 
their  large-screen  situational  display  system  in  their  main  operations  center. 
NEADS  mission  is  to  provide  Total  Force  Air  Defense  and  threat  warning  to 
North  America  through  readiness,  detection  and  identification,  and  if  necessary, 
force  application.  For  several  months,  our  team  worked  very  closely  with  NEADS 
to  integrate  a  large-screen  display,  based  on  DataWall  technology,  designed 
specifically  for  their  facility,  budget,  and  demanding  around-the-clock  operational 
needs.  This  in  turn  has  provided  the  foundation  for  potential  integration  of  other 
Information  Directorate  Situation  Awareness  (SA)  tools  and  techniques  that  might 
apply  to  the  NEADS  mission. 
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5.4.1  Requirements 


The  display  system  to  be  replaced  consisted  of  five  LCD-based  front  projectors 
each  at  a  resolution  of  1024  x  768  pixels  displayed  on  five  separate  6'  x  4' 
screens  approximately  10'  off  the  ground.  The  problems  the  NEADS  staff  was 
experiencing  with  the  configuration  included  overheating  and  subsequent 
permanent  damage  to  their  projectors  from  excessive  use.  The  projectors  were 
also  not  mounted  to  any  permanent  fixture  that  caused  drastic  geometry 
distortions  to  the  projected  images  as  a  result  of  the  inability  to  position  the 
projectors  at  the  correct  pitch  relative  to  the  screen.  The  screens  were  mounted 
at  a  slight  forward  pitch  to  compensate  to  a  certain  extent,  but  still  produced  a 
less  than  optimal  view  for  operators  at  an  elevated  position  at  the  back  of  the 
room  in  the  Battlecab  (BC). 

The  new  display  had  to  be  designed  for  effective  viewing  by  operators  on  the 
main  floor  as  well  as  from  the  elevated  position  of  the  BC.  The  system  needed  to 
operate  around  the  clock  without  fail  24-hours/day,  7-days/week.  Input  devices 
needed  to  be  effective  from  the  BC  primarily,  with  a  certain  degree  of  interaction 
if  possible  from  the  Ops  floor.  The  budget  for  new  equipment  was  extremely 
limited. 


5.4.2  System  Description  and  Installation 

One  element  for  consideration  in  the  new  system  design  was  the  physical 
characteristics  of  the  facility.  The  unconventional  architectural  elements  of  the 
room  such  as  the  extremely  high  curved  ceiling  and  the  shear  size  of  the  room  at 
approximately  50'  x  50'  posed  significant  challenges.  The  installation  of  the 
system  also  had  to  be  accomplished  with  no  disruption  to  ongoing  24-hour 
operations. 

It  was  primarily  cost  and  space  considerations  that  led  to  a  large  front  projection 
approach.  The  dim  lighting  conditions  were  also  beneficial  to  this  approach.  The 
tall  ceiling  ensured  adequate  height  for  mounting  the  screen  and  projectors 
above  operators’  heads  that  might  otherwise  cast  shadows  on  the  screen. 

A  single  30'  x  8'  front  projection  screen  was  mounted  as  high  as  the  arched 
ceiling  would  allow;  approximately  7/4'  off  the  ground.  Three  LCD-based 
projectors  each  with  a  resolution  of  1280  x  1024  pixels,  with  long-throw  lenses 
were  mounted  to  permanent,  but  adjustable  fixtures  above  the  BC.  The  mounts 
allowed  mechanical  adjustment  to  the  projectors  to  create  a  seamless  image 
across  the  entire  display  screen.  The  projectors  selected  had  the  desired 
resolution  and  lenses  available  with  the  correct  focal  length  for  the  throw  distance 
and  image  size.  The  need  for  around-the-clock  operation  necessitated  an 
additional  bank  of  three  projectors  be  incorporated  into  the  design  to  take  half  of 
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the  24-hours/day,  7-days/week  operational  load  at  six-hour  intervals.  It  also 
provides  redundancy  in  the  event  of  projector(s)  failure.  The  projectors  also  had 
a  built-in  motorized  horizontal  and  vertical  lens  shift  and  programmable  timers. 
The  lens  shift  feature  allowed  the  projector  pairs  to  be  mounted  next  to  each 
other  and  their  respective  images  shifted  horizontally  to  the  same  area  of  the 
display  screen.  When  either  was  powered  on  it  would  properly  align  with  the 
image  next  to  it.  The  built  in  timers  allowed  each  projector  to  be  programmed 
with  its  allotted  duty  cycle.  Days  of  the  week  and  six-hour  operation  times  to 
power  on  and  off  could  be  set.  The  six  projector  layout  was  also  specifically 
designed  to  allow  uninterrupted  display  functionality  during  projector  or  lamp 
replacement  (Figure  29). 


46.6' 


Figure  29  NEADS  Projector  Configuration 


A  single  PC  server  configured  with  three  video  cards  drives  the  display. 
Conventional  keyboard  and  mouse  in  front  of  a  three-screen  display  console 
allows  interaction  with  applications  running  on  the  system  in  the  BC.  Wireless  RF 
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mice  with  100'  range  can  also  be  used  inside  the  BC  or  anywhere  on  the  Ops 
floor.  The  entire  installation  was  completed  November  2002. 

The  technology  advanced  was  the  development  and  installation  of  a  DataWall  in 
an  operational  command  and  control  environment  on  a  much  larger  scale  than 
what  has  been  previously  accomplished  in  the  lab.  One  designed  to  be 
economical,  with  redundancy  to  ensure  minimum  downtime,  and  with  effective 
modes  of  input  for  operators. 


5.4.3  Lessons  Learned 

Although  the  system  remained  operational  24  hours/  day,  7  days/week  for  more 
than  2  years  they  did  experience  some  degree  of  failures.  At  the  time  of  the 
installation  display  devices  designed  for  around-the-clock  operations  were 
extremely  limited,  very  costly,  and  would  have  required  significant  construction  of 
a  support  structure  to  install  a  system  of  comparable  size  and  resolution.  There 
were  no  commercially  available  projectors  that  could  operate  for  such  extended 
periods  of  time  without  eventual  image  degradation  and  would  be  covered  by  the 
factory  warranties  if  they  failed.  LCD-based  projectors  require  very  bright  lamps 
with  subsequent  heat  that  eventually  started  to  degrade  the  color  balance  of  the 
projected  images  due  to  damage  to  the  LCDs.  The  six  hour  operating  intervals 
helped,  but  the  12  hour/day  total  duty  cycle  for  each  projector  was  still  too  taxing. 
The  projected  images  did  degrade  over  time,  but  were  always  usable  and  never 
completely  failed.  Spare  projectors  were  purchased  and  installed  so  that  the 
damaged  projectors  could  be  repaired  by  the  vendor. 

There  were  also  issues  with  the  3-output  DataWall  server  that  was  installed  to 
drive  the  display.  It  was  later  determined  that  three  input/output  sources  was  not 
sufficient.  The  noise  of  the  DataWall  server  fans  was  also  an  issue  and  could  not 
be  placed  in  a  location  where  operations  were  being  run.  However,  it  was 
important  for  the  equipment  to  be  placed  next  to  a  user  in  order  to  manipulate  the 
input/output  signals. 

Digital  Light  Processing™  (DLP®)  technology  which  is  becoming  ever  more 
prevalent  in  the  consumer  and  professional  display  community  has  emerged  as 
the  de  facto  standard  for  control  room  24/7  operational  environments.  They 
generate  significantly  less  heat  and  because  the  technology  is  reflective  rather 
than  transmissive,  has  a  lower  failure  rate  than  an  LCD-based  system. 
Manufacturers  are  offering  DLP®-based  products  that  are  rated  and  warranted 
for  24/7  operation.  At  the  time  of  the  NEADS  installation  products  available  with 
DLP®  technology  were  very  limited,  expensive,  only  in  smaller  stackable  cube 
displays,  and  with  limited  resolution.  The  budget  set  for  the  project  could  not 
afford  such  an  installation. 
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5.4.4  Impact  and  Current  Status 


In  August  of  2005,  NEADS  hired  a  contractor  to  upgrade  a  majority  of  the 
DataWall  system  that  was  problematic,  which  included  the  server  and  projectors 
that  the  ADII  team  initially  installed  with  a  new  AudioA/isual  (AA/)  system. 

The  projectors  were  replaced  with  Panasonic  model  PT-D7600U  that  are  3-chip 
DLP  projectors  with  a  liquid-cooled  light  engine  and  sealed  optics  system  offering 
longer  projector  life.  Although  they  are  not  specifically  rated  for  24/7  operations, 
the  technology  characteristics  of  DLPs,  along  with  the  sealed-liquid-cooling 
system,  and  operating  at  a  lower  brightness  setting  should  prove  to  be  a  more 
reliable  system  for  the  environment. 

The  DataWall  server  was  replaced  with  an  AMX  Master  Controller  model  Nl 
3000+  and  Extron  Matrix  Switcher,  which  gives  the  ability  to  have  32- 
inputs/outputs  for  the  system.  They  also  purchased  two  touch-screen  control 
panels  that  can  remotely  control  the  inputs  and  outputs  of  the  system. 

They  also  installed  eight  42"  LCD  screens  and  four  60"  LCD  screens  to  give 
more  flexibility  in  displaying  operational  information.  This  gives  the  Ops 
personnel  more  display  capability. 

Their  display  needs,  budget,  and  display  technology  changed  dramatically  from 
the  time  the  original  installation  was  accomplished.  It  is  encouraging  that  they  are 
now  able  get  the  customer  support  they  need  that  we  are  not  adequately  staffed 
to  provide  and  are  able  to  take  advantage  of  the  latest  display  technology  better 
suited  for  their  high  tempo  24/7  C2  environment.  It  is  a  little  disappointing  that 
they  have  in  certain  respects  taken  a  few  steps  backward  in  terms  of  managing 
the  display  space  by  reverting  to  a  conventional  video  switching  system.  This  fact 
does  reinforce  the  requirement  to  further  improve  the  usability  of  the  DataWall, 
however. 


5.5  Air  Force  Institute  of  Technology,  Wright-Patterson  AFB  OH 

The  Air  Force  Institute  of  Technology  (AFIT)  and  Wright  State  University  (WSU) 
received  a  grant  from  the  Ohio  Board  of  Regents  to  develop  a  collaborative 
virtual  environment  for  exploring  HCI  and  visualization  techniques.  AFIT’s 
portion  of  this  grant  involved  the  acquisition  of  a  display  system  for  command 
and  control  visualization.  Having  seen  the  AFRL/IF  developed  Portable 
DataWall,  AFIT  sought  to  acquire  one  of  their  own  to  have  a  configuration  that  is 
common  to  AFRL  and  other  AF  organizations  to  better  facilitate  their  research  in 
support  of  these  organizations.  Despite  AFIT’s  limited  budget  to  purchase  a 
display  system,  AFRL/IF  agreed  to  provide  a  slightly  used  Portable  DataWall  at  a 
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significantly  reduced  cost  in  exchange  for  some  research  in  support  of  the 
DataWall  project. 

A  CRADA  was  established  among  AFRL/IF,  AFIT,  and  Wright  State  University 
(WSU).  The  objective  of  the  agreement  is  to  share  HCI,  visualization,  and 
display  system  technology  under  development  at  AFRL/IF,  AFIT,  and  WSU. 
AFRL/IF  delivered  and  installed  a  Portable  Interactive  DataWall  at  AFIT  on  30 
March  2004.  WSU,  using  funds  from  the  grant,  partially  reimbursed  AFRL/IF  for 
the  cost  of  the  DataWall.  AFIT  and  WSU  will  research  HCI  and  visualization 
techniques  using  the  DataWall,  sharing  meaningful  results  with  AFRL/IF. 

Anticipated  benefits  to  AFRL/IF  from  this  agreement  include  improvements  to  the 
DataWall’s  HCI  capabilities  either  from  evaluation  feedback  or  new  software 
developed  by  AFIT.  In  addition,  AFIT  will  deliver  C2  visualization  software  to 
AFRL/IF  that  has  been  successfully  installed  on  the  DataWall  and  determined  to 
be  useful  to  AFRL/IF  and  further  show  the  utility  of  a  large  interactive  display  in  a 
C2  environment.  It  was  also  an  opportunity  for  someone  to  make  use  of  what  we 
considered  obsolete  equipment  in  return  for  some  funds  to  purchase  more 
current  equipment  to  further  the  DataWall  in-house  development. 


5.5.1  Impact  and  Current  Status 

The  DataWall  installation  at  AFIT  is  still  operational  and  includes  the  addition  of 
an  InterSense  IS-900  virtual  reality  controller.  Since  arriving  at  AFIT,  the 
DataWall  has  been  used  to  demonstrate  and  interact  with  two  different  UAV 
swarm  models,  and  also  in  the  initial  interfacing  with  the  InterSense  controller. 

Some  of  the  usage  hurdles  that  have  been  identified  include  no  mechanism  for 
multiple  users  to  interact  with  the  DataWall  and  information  represented 
simultaneously.  Providing  the  capability  for  multiple  users  to  simultaneously 
interact  with  the  display  is  an  ongoing  research  project  that  has  been  highly 
successful  since  the  initial  DataWall  installation  at  AFIT,  but  is  not  a  capability 
currently  installed  at  their  site.  It  was  not  part  of  the  original  agreement  so  there 
were  no  plans  to  provide  it.  Their  needs  are  slightly  different,  however,  in  that 
they  would  like  multiple  users  to  be  able  to  interact  within  a  single  application 
rather  than  multiple  applications  as  our  current  system  has  been  designed.  In 
most  instances  this  is  not  very  useful  in  our  opinion  so  would  most  likely  require 
custom  applications  to  support  the  capability.  There  are  currently  no  plans  to  do 
so,  but  have  recognized  there  is  a  need,  albeit  limited,  with  potential  for  future 
development. 

Also  a  human  caused  distance/accuracy  problem  was  identified,  in  that  it  is 
extremely  difficult  to  use  the  laser  pointer  accurately  at  distances  where  the 
entire  wall  is  in  view  or  larger.  With  the  additional  interface  to  the  InterSense 
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controller  being  developed,  they  anticipate  that  this  will  become  more 
manageable  but  may  also  open  up  a  large  number  of  additional  technical  hurdles 
associated  with  distance  and  perspective.  The  problem  has  been  a  continuing 
criticism  we  hope  to  be  able  to  solve  or  ease  at  some  point  in  the  near  future. 


6  Future  Research 


6. 1  Data  Wall  Display  Enhancements 

The  importance  of  an  expanded  viewable  space  and  increased  image  resolution, 
are  imperatively  considered  in  the  designs  of  the  next  generation  Interactive 
DataWall,  and  other  display  system  development.  Because  of  the  often  close 
screen  interaction  with  these  displays,  a  pixel  density  that  matches  the  visual 
acuity  of  the  human  eye  at  these  close  distances  is  desired.  As  COTS  display 
technology  improves  in  both  the  direct-view  and  projector  product  markets  they 
will  be  evaluated  and  potentially  integrated  into  our  experimental  systems.  The 
recent  display  technology  push  to  support  the  emerging  high-definition  television 
(HDTV)  standard  has  resulted  in  a  much  wider  selection  of  products  available 
that  meet  our  display  design  requirements.  Among  the  planned  enhancements 
include  incorporating  multiple  HD  resolution  devices  (1920x1080)  and  in  larger 
matrix  configurations. 


6.2  Touch  Screen  Integration  Completion 

Touch  screen  technology  has  improved  significantly  over  the  past  few  years 
particularly  with  respect  to  scalability  and  image  quality  preservation.  The  benefit 
of  adding  touch  interaction  to  the  IDW  is  being  explored.  Touch  technology  would 
not  be  a  replacement  for  the  laser  pointer  interaction  but  rather  another 
enhancement  by  providing  the  users  with  another  mode  of  input. 

Integration  of  a  touch  screen  capability  for  the  DataWall  is  well  underway  with  the 
CIDW  frame  redesign,  fabrication  and  delivery  of  a  prototype  system  complete.  A 
screen  design  will  be  finalized  and  integrated  into  the  system  shortly.  Confidence 
is  high  in  the  ability  to  scale  touch  screen  technology  to  large  multi-projector 
displays  with  recent  advances  in  the  state-of-the-art.  The  feasibility  of  utilizing  a 
common  set  of  cameras  to  track  lasers  and  touch  will  also  be  investigated. 

Based  on  the  success  of  the  integration  of  touch  interaction  into  the  CIDW  and 
human  factors  evaluations  for  this  type  of  interaction  on  very  large  displays  will 
dictate  the  degree  to  which  it  will  be  integrated  into  future  portable  and  built-in 
DataWall  systems. 
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6.3  Gesture  Recognition 


Gesture  recognition  geared  for  a  large  screen  display  is  being  explored  and 
developed  through  a  research  program  with  Howard  University.  The  central 
approach  for  the  feasibility  of  the  program  will  utilize  a  differentiating  technique  to 
initially  identify  pointers  via  image  triangulation  methods  for  accurate  positioning 
and  tracking  of  pointers,  and  the  usage  of  cutting  edge  high  resolution  cameras. 
Methods  for  multiple  pointers  and  a  working  system  for  recognizing  a  variety  of 
hand  gestures  will  be  developed  during  the  course  of  the  research  program's 
duration. 

As  an  initial  stage  for  evaluating  the  feasibility  of  tracking  pointers,  a  simulation 
for  pointer  detection  in  3-dimensional  space  for  validating  the  accuracy  and 
performance  of  the  algorithm  was  conducted.  Preliminary  results  hypothetically 
revealed  significant  figures  for  tracking.  The  exploration  and  vision  of  such 
capability  for  an  extensively  scaled  display  environment  leads  to  direct 
applicability  for  multiple  participants  to  interact  with  a  large  screen  display  in  a 
Command  and  Control  setting  with  minimal  or  no  obstruction. 


6.4  Portable  Display  Modules  (PDMs) 

One  enhanced  capability  expressed  by  users  of  the  Portable  DataWall  systems 
is  a  desire  for  a  larger  display  system  that  is  still  portable.  Although  the  9'  width 
has  proven  adequate,  the  height  of  29"  is  limiting.  In  order  to  keep  the 
operational  depth  from  increasing  and  avoiding  the  use  of  folded  optics,  the 
approach  planned  is  to  tile  projectors  vertically  as  well  as  horizontally.  A 
preliminary  design  for  a  Portable  Display  Module  (PDM)  has  been 
conceptualized  that  incorporates  two  projectors  stacked  vertically  and  a  screen 
36"W  x  54"H.  The  screen  will  be  a  commercially  available  near  frameless  system 
that  will  allow  multiple  PDMs  to  be  linked  together  to  build  any  n  x  2  tiled  display 
that  is  54"  high  and  virtually  seamless  (Figure  30). 


Figure  30  Portable  Display  Module  Concept  (3  module  example) 
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6.5  Automatic  Projector  and  Camera  Alignment 

Developing  and  incorporating  an  automatic  projector  and  camera  alignment 
system  to  help  speed  the  set-up  of  the  IDW  is  being  investigated.  It  will  be  of 
particular  value  to  the  portable  systems  being  developed  that  require  realignment 
after  each  deployment.  The  typical  user  of  the  IDW  is  not  an  Audio/Visual  expert 
and  it  is  unrealistic  in  most  instances  to  expect  this  user  to  arrive  at  a  nearly 
seamless  alignment  of  the  projectors.  A  very  daunting  task  will  be  to  develop  an 
electromechanical  projector  mount  that  uses  optical  sensors  to  determine 
alignment  accuracy  and  automatically  correct  the  projector  position.  The  intent  is 
to  still  rely  on  physical  repositioning  of  the  projectors  to  correct  the  image 
geometry  rather  than  digitally.  Edge  blending  and  image  warping  techniques 
have  been  avoided  in  the  past  to  preserve  the  full  resolution  of  the  display. 
However,  it  will  be  investigated  for  the  final  fine-tune  adjustment  to  further 
improve  image  quality  and  simplify  set-up  for  inexperienced  operators. 
Furthermore,  it  is  unrealistic  to  expect  a  novice  user  to  tackle  every  possible 
scenario  in  which  the  cameras  are  the  cause  of  any  laser  tracking  problems. 
Methods  for  simplifying  camera  positioning  and  improvements  to  the  calibration 
process  are  also  being  investigated. 


7  Conclusions 

The  Interactive  DataWall  in  its  various  stages  of  development  and  configurations 
has  been  the  all  encapsulating  operational  product  that  has  resulted  from  all  of 
the  ADII  display  and  HCI  research.  It  has  been  demonstrated  to  a  number  of 
visitors  at  AFRL/IF  at  Rome  Research  Site.  The  Deployable  and 
Portable/Collapsible  Interactive  DataWalls  have  been  deployed  and 
demonstrated  at  a  number  of  off-site  conferences  and  military  exercises.  Several 
Portable  DataWalls  have  been  transitioned  to  customers  outside  our  research 
facility.  They  have  been  extremely  well  received,  so  the  ADII  team  is  confident 
that  research  efforts  are  headed  in  the  right  direction  for  next  generation  C2 
systems.  Although  a  considerable  amount  of  research  and  development  remains, 
extremely  capable  systems  have  been  created,  integrating  a  significant  amount 
of  commercially  available  hardware  and  software.  Its  evolution  has  been  driven 
by  warfighter  requirements  and  valuable  field  experiences. 
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